tinyq: 一个任务队列框架

最近尝试着写了一个简单的任务队列框架,整体的功能类似 celery, rq 之类的任务队列框架。 下面简单的介绍一下。

安装

安装:
pip install tinyq

这个框架使用 redis 作为队列服务,所以同时也需要安装 redis 服务程序。

Hello World

  1. 启动 redis 服务:

    $ redis-server
    
  2. 增加一个 app.py 文件:

    from tinyq import Application
    
    app = Application()
    
    
    @app.task()
    def add(m, n):
        return m + n
    
  3. 通过 python shell 增加几个 delay job:

    from app import app
    
    for m in range(10):
     for n in range(3):
         add.delay(m, n)
    
  4. 启动 worker 消费队列中的 job:

    $ tinyq -l info
    2017-03-12 21:27:12,322 - WARNING - tinyq.runner[line:73 thread:MainThread(140736379601856) process:MainProcess(15388)] - Starting TinyQ worker, version 0.1.0...
    2017-03-12 21:27:12,446 - INFO - tinyq.worker[line:65 thread:Worker-2(123145554059264) process:MainProcess(15388)] - Got a job: <Job: id: 9687d9dd-30f4-4920-bd0c-924e672d9794, task_name: add>
    2017-03-12 21:27:12,447 - INFO - tinyq.worker[line:67 thread:Worker-2(123145554059264) process:MainProcess(15388)] - Finish run job <Job: id: 9687d9dd-30f4-4920-bd0c-924e672d9794, task_name: add>
    2017-03-12 21:27:12,500 - INFO - tinyq.worker[line:65 thread:Worker-5(123145569824768) process:MainProcess(15388)] - Got a job: <Job: id: 315f4ead-cedb-4b7a-b3c6-d328b0152e35, task_name: add>
    2017-03-12 21:27:12,501 - INFO - tinyq.worker[line:67 thread:Worker-5(123145569824768) process:MainProcess(15388)] - Finish run job <Job: id: 315f4ead-cedb-4b7a-b3c6-d328b0152e35, task_name: add>
    2017-03-12 21:27:12,610 - INFO - tinyq.worker[line:65 thread:Worker-1(123145548804096) process:MainProcess(15388)] - Got a job: <Job: id: a014ee87-0200-4b78-af25-6fe8dcca3f14, task_name: add>
    2017-03-12 21:27:12,610 - INFO - tinyq.worker[line:67 thread:Worker-1(123145548804096) process:MainProcess(15388)] - Finish run job <Job: id: a014ee87-0200-4b78-af25-6fe8dcca3f14, task_name: add>
    ^C2017-03-12 21:27:13,863 - WARNING - tinyq.runner[line:144 thread:MainThread(140736379601856) process:MainProcess(15388)] - Received stop signal, warm shutdown...
    2017-03-12 21:27:13,886 - WARNING - tinyq.runner[line:135 thread:Worker-2(123145554059264) process:MainProcess(15388)] - Exit worker Worker-2.
    2017-03-12 21:27:13,896 - WARNING - tinyq.runner[line:135 thread:Worker-7(123145580335104) process:MainProcess(15388)] - Exit worker Worker-7.
    2017-03-12 21:27:13,906 - WARNING - tinyq.runner[line:135 thread:Scheduler(123145538293760) process:MainProcess(15388)] - Exit worker Scheduler.
    2017-03-12 21:27:13,924 - WARNING - tinyq.runner[line:135 thread:Worker-5(123145569824768) process:MainProcess(15388)] - Exit worker Worker-5.
    2017-03-12 21:27:13,936 - WARNING - tinyq.runner[line:135 thread:Worker-0(123145543548928) process:MainProcess(15388)] - Exit worker Worker-0.
    2017-03-12 21:27:13,956 - WARNING - tinyq.runner[line:135 thread:Worker-4(123145564569600) process:MainProcess(15388)] - Exit worker Worker-4.
    2017-03-12 21:27:13,978 - WARNING - tinyq.runner[line:135 thread:Worker-6(123145575079936) process:MainProcess(15388)] - Exit worker Worker-6.
    2017-03-12 21:27:14,017 - WARNING - tinyq.runner[line:135 thread:Worker-1(123145548804096) process:MainProcess(15388)] - Exit worker Worker-1.
    2017-03-12 21:27:14,068 - WARNING - tinyq.runner[line:135 thread:Worker-3(123145559314432) process:MainProcess(15388)] - Exit worker Worker-3.
    2017-03-12 21:27:14,068 - WARNING - tinyq.runner[line:101 thread:MainThread(140736379601856) process:MainProcess(15388)] - Exit workers.
    $
    

功能

通过上面的 Hello World 示例可以看出 tinyq 的功能类似 rq/celery。 还有一些 Hello World 中没有展示出来的功能:

  • app = Application(), 可以给 Application 传递一个参数来指定 redis 配置: app = Application('redis://192.168.99.100:2375/1')

  • app.py 也可以叫其他的名字,只要在启动 worker 的时候指定 --app 参数就可以了。 比如有如下文件:

    $ cat test/hello.py
    
      tinyq_app = Application()
    

    那么 --app 参数的值就是 test.hello.tinyq_app , 关键点是告诉 worker Application 实例所在位置

  • 装饰器 @app.task() 还有个 name 参数,用于指定 task 的名称(默认是被装饰的函数的名称)

  • 既可以通过 add.delay 把操作放到队列中,也可以通过 add(m, n) 立即执行操作

  • 通过 control + c 退出 worker 时会等待正在处理的 job 全部完成后再退出

  • worker 还支持其他参数:

    $ tinyq -h
    usage: tinyq [-h] [-V] [-u URI] [-v] [-w WORKER_NUMBER] [-a APP]
                 [-l {debug,info,warn,error,critical}]
    
    Starts a TinyQ worker.
    
    optional arguments:
      -h, --help            show this help message and exit
      -V, --version         show program's version number and exit
      -u URI, --uri URI     The Redis URI (default: redis://)
      -v, --verbose         Show more output
      -w WORKER_NUMBER, --worker-number WORKER_NUMBER
                            Worker number (default: 8)
      -a APP, --app APP     Application path (default: app.app)
      -l {debug,info,warn,error,critical}, --log-level {debug,info,warn,error,critical}
                            Logging level (default: warn)
    

That's all.

有任何意见欢迎在下方留言或者加我微信交流


Comments

comments powered by Disqus