iSt0ne's Notes

CentOS 6.3 Gearman环境搭建

Gearman是最早由LiveJournal内部开发并使用的一个通用并行任务调度框架,允许不同语言直接通过非常简单的方式进行互操作。前台提交工作任务(Task)和参数,由后台工作进程(Worker)完成实际工作。

##安装gearman job server

安装gearman依赖包:

1
yum install boost-devel curl libcurl-devel uuid-devel libevent-devel

注意:boost的版本应该不低于1.37.0

下载最新版本的gearmand安装:

1
2
3
4
wget https://launchpad.net/gearmand/1.2/1.1.5/+download/gearmand-1.1.5.tar.gz
./configure --prefix=/usr/local/gearman
make
make install

启动gearman:

1
2
mkdir -p /usr/local/gearman/var/log/
/usr/local/gearman/sbin/gearmand -d

##安装gearman php扩展:

安装php和-ImageMagick:

1
2
3
4
5
yum install httpd php ImageMagick php-imagick
http://pecl.php.net/get/gearman-1.1.1.tgz
tar -xzvf gearman-1.1.1.tgz
cd gearman-1.1.1

我用php5.2.14安装的时候发现出现E_DEPRECATED未定义错误,这个错误等级变量是5.3更新的所以5.2的php装不上这个扩展
有两种办法解决这个问题

  1. 升级php到5.3版本
  2. 替换php_gearman.c中的E_DEPRECATED为其他错误等级变量(有两处),比如E_ERROR
    1
    2
    3
    4
    5
    /usr/local/sinasrv2/bin/phpize
    sed -i 's/E_DEPRECATED/E_ERROR/g' php_gearman.c
    ./configure --with-gearman=/usr/local/gearman/ --with-php-config=/usr/local/sinasrv2/bin/php-config
    make
    make install

php gearman客户端文档参考:http://www.php.net/manual/zh/book.gearman.php

##安装python扩展:

如果没有安装python setuptools,根据自己的python版本下载对应的软件包。

1
2
wget https://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c11-py2.6.egg#md5=bfa92100bd772d5a213eedd356d64086
> sh setuptools-0.6c11-py2.6.egg

安装python的gearman扩展

1
2
3
4
wget https://pypi.python.org/packages/source/g/gearman/gearman-2.0.1.tar.gz
tar xzvf gearman-2.0.1.tar.gz
cd gearman-2.0.1
python setup.py install

Python gearman客户端文档的参考:http://pythonhosted.org/gearman/index.html

##测试:

Gearman Server:10.207.0.191(Gearman 191) 10.207.0.192(Gearman 192)
Gearman Client:10.207.0.191
Gearman Worker:10.207.0.192

1). 在10.207.0.191和10.207.0.192上启动Gearman服务:

1
/usr/local/gearman/sbin/gearmand -d

2). 在10.207.0.191上部署Gearman Client脚本:
Python客户端脚本(client.py)如下:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
#!/usr/bin/env python
# -*- coding:utf-8 -*-
# This is client.py
from gearman import GearmanClient
new_client = GearmanClient(['10.207.0.191:4730','10.207.0.192:4730'])
current_request = new_client.submit_job('echo', 'foo')
new_result = current_request.result
print new_result
```
3). 在10.207.0.192上部署Gearman Worker脚本:
Python客户端脚本(worker.py)如下:
```python
#!/usr/bin/env python
# -*- coding:utf-8 -*-
# This is worker.py
import os
import gearman
import math
class CustomGearmanWorker(gearman.GearmanWorker):
def on_job_execute(self, current_job):
print "Job started"
return super(CustomGearmanWorker, self).on_job_execute(current_job)
def task_callback(gearman_worker, job):
data="%s\n" %job.data
return data
new_worker = CustomGearmanWorker(['10.207.0.191:4730','10.207.0.192:4730'])
new_worker.register_task("echo", task_callback)
new_worker.work()
```
PHP客户端脚本(worker.php)如下:
```php
<?php
$worker= new GearmanWorker();
$worker->addServers("10.207.0.191:4730,10.207.0.192:4730");
$worker->addFunction("resize", "my_resize_function");
while ($worker->work());
function my_resize_function($job)
{
$thumb = new Imagick();
$thumb->readImageBlob($job->workload());
if ($thumb->getImageHeight() > 600)
$thumb->scaleImage(0, 600);
else if ($thumb->getImageWidth() > 800)
$thumb->scaleImage(800, 0);
return $thumb->getImageBlob();
}
?>

4). 在10.207.0.192上启动python worker和php worker
执行nohup python worker.py &命令,启动1个python worker
执行两次nohup /usr/bin/php worker.php &命令,启动2个php worker
5). 在10.207.0.191上执行
执行client.py脚本,返回foo:

1
2
python client.py
foo

把一个2560 × 1600图片转换成一个800 x 600的图片:

1
2
3
4
/usr/local/gearman/bin/gearman -h 10.207.0.192 -p 4730 -f resize <full.png >small.png
ll full.png small.png
-rw-r--r-- 1 root root 513870 2月 21 19:37 full.png
-rw-r--r-- 1 root root 165084 2月 22 18:39 small.png

参考:http://gearman.org/getting_started