So Tired !_! 逆水行舟, 不进则退!

29Feb/12

MetaTrader & Automated Trading & Expert Advisors Best Trading platform for Forex trading

Posted by Nick Xu

Configuration at Startup

The client terminal can be launched with some predefined settings. For this purpose, the configuration file name will be passed to the client terminal as a parameter.

For example:

Code:
terminal.exe configstart.ini
  terminal.exe test1.txt
  terminal.exe "C:Program FilesMetaTrader 4configsettings25.ini"

If the full path to the file (Drive:SubDirectoryFileName) is not given, the file will be searched for in the client terminal directory. The configuration file contains lines of the following appearance:
[Parameter] = [Value]

Comments start with a semicolon ( and are not processed.

The configuration file parameters can be divided into several groups: common settings, proxy server settings (the "Server" tab in the terminal settings), FTP settings (the "Publisher" tab in the server settings), EA settings (the "Expert Advisors" tab in the server settings), the expert or script single-launch settings, settings of the Strategy Tester launch.

Common Settings
Profile - the subdirectory name in the /profiles directory. The charts will be opened in the client terminal according to the given profile. If this parameter is not specified, the current profile will be opened.

MarketWatch - file name (the symbolsets directory) that contains the symbol list to be shown in the Market Watch window. A file like this can be obtained using the window context menu command of the "Market Watch - Sets - Save As...".

Login - the number of the account to connect to at startup. If this parameter is not specified, the current login will be used.

Password - the password that allows entering the system. This parameter will be ignored if the client terminal stores personal data on the disk and the account to be connected is in the list.

Server - the name of the trade server to be connected to. The server name is the same as the name of the corresponding .srv file stored in the /config directory. This patameter will be ignored if the information about the account to be connected was stored on the disk.

AutoConfiguration - "true" or "false" depending on whether the autoconfiguration setting should be enabled or not. If this parameter is not specified, the value from the current server settings will be used.

DataServer - address of the data center. This record can be ignored if the server autoconfiguration s enabled. If this parameter is not specified, the value from the current server settings will be used.

EnableDDE - "true" or "false" depending on whether DDE server should be enabled or not. If this parameter is not specified, the value from the current server settings will be used.

EnableNews - "true" or "false" depending on whether receiving of news should be allowed or not. If this parameter is not specified, the value from the current server settings will be used.

Example:

Code:
  ; common settings
  Profile=test 3
  MarketWatch=set2.set
  Login=12345
  Password=xxxxxx
  Server=MetaQuotes-demo
  AutoConfiguration=false
  DataServer=192.168.0.1:443
  EnableDDE=true
  EnableNews=false

Proxy Server Settings

ProxyEnable - "true" or "false" depending on whether or not a proxy server should be used for connection to the trade server.

ProxyServer - proxy server address.

ProxyType - proxy server type.It can be "HTTP", "SOCKS4", or "SOCKS5".

ProxyLogin - login to be authorized on proxy server.

ProxyPassword - password to access to proxy server.

If any of the above parameters are not specified, the current settings of the client terminal are used (proxy settings in the "Server" tab of the client temrinal settings).

Example:

Code:
 ; proxy settings
  ProxyEnable=true
  ProxyServer=proxy.company.com:3128
  ProxyType=HTTP
  ProxyLogin=user45
  ProxyPassword=xxx

FTP Settings
FTPEnable - enable/disable publishing. The possible values are "true" or "false".

FTPPassiveMode - enable/disable the passive mode of data transfer. The possible values are "true" or "false".

FTPAccount - the number of the account the state of which to be sent to the FTP.

FTPServer - FTP server address.

FTPLogin - the login for authorization on the FTP server.

FTPPassword - the password to access to the FTP server.

FTPPath - the name of the FTP server directory in which the report is placed.

FTPPeriod - the periodicity, in minutes, of the reporting to the FTP server.

If any of the above-listed parameters are not specified, the current client terminal settings are used (the "Publisher" tab in the server settings).

Example:

Code:
; ftp settings
  FTPEnable=true
  FTPPassiveMode=false
  FTPAccount=12345
  FTPServer=ftp.company.com
  FTPLogin=admin
  FTPPassword=pAssWOrd123
  FTPPath=/inetpub
  FTPPeriod=10

EA Settings
ExpertsEnable - enable/disable experts.

ExpertsDllImport - enable/disable DLL imports.

ExpertsDllConfirm - enable/disable manual confirmation of DLL-functions calling.

ExpertsExpImport - enable/disable import of functions from external experts or MQL4 libraries.

ExpertsTrades - enable/disable the experts trading.

ExpertsTradesConfirm - enable/disable manual confirmation of the trade operation performed by the expert.

Quote:
Attention: All parameters of an experts group can take values of either "true" or "false".

If any of the above-listed parameters is not specified, the current client terminal settings will be used (the "Expert Advisors" in the server settings).

Example:

Code:
 ; experts settings
  ExpertsEnable=true
  ExpertsDllImport=true
  ExpertsDllConfirm=true
  ExpertsExpImport=true
  ExpertsTrades=true
  ExpertsTradesConfirm=false

The Expert and/or Script Single-Launch Settings
Symbol - the symbol of the security the chart of which should be opened immediately after the terminal startup. After the client terminal has been closed, the information about this extra chart is not saved. At the terminal restart, without the configuraiton file, this chart will not be opened. If this parameter is not specified, no extra chart will be opened.

Period - the chart timeframe (M1, M5, M15, M30, H1, H4, D1, W1, MN). If this parameter is not specified, H1 is used.

Template - the name of the templatre file (the templates directory), which should be applied to the chart.

Expert - the name of the expert that should be launched after the client terminal has started. The expert is launched in the chart, which has been opened according to the data specified in Symbol and Period. If the Symbol parameter has not been not specified, no extra chart opens, and the expert will be launched in the first chart of the current profile. If there are no charts in the current profile, the expert will not be launched. If this parameter has not been specified, no expert is launched.

ExpertParameters - the name of the file containing the expert parameters (the expertspresets directory). This file can be created in the expert properties window by pressing of the "Inputs - Save" button. It is normally used to save the inputs other than the default ones. If this parameter has not been specified, the default inputs are used.

Script - the name of the script, which must be launched after the client terminal startup. The script is launched according to the same rules that are eligible for the expert (described above).

ScriptParameters - the name of the file containing the script parameters (the expertspresets directory). This file is made in the same way as that for the expert.

Example:

Code:
 ; open chart and run expert and/or script
  Symbol=EURUSD
  Period=H4
  Template=popular.tpl
  Expert=MACD Sample
  ExpertParameters=macd.set
  Script=period_converter
  ScriptParameters=

Settings of the Strategy Tester Launch
TestExpert - the name of the expert to be launched for testing. If this parameter has not been specified, no testing is launched.

TestExpertParameters - the name of the file containing parameters (the tester directory). The file can be created in the Properties window of the expert under test by clicking the "Inputs - Save" button. It is normally used to save parameters other than the default ones. Other parameters of the expert under test in the "Testing" and "Optimization" tabs (as well as in the "Inputs" tab if this parameter has not been specified) are filled up with the values automatically saved in the tester[the expert name].ini file after the latest test.

TestSymbol - the name of the symbol used for the expert testing. If this parameter has not been specified, the latest value used in the tester is used.

TestPeriod - the chart period (M1, M5, M15, M30, H1, H4, D1, W1, MN). If this parameter has not been specified, H1 is used.

TestModel - 0, 1, or 2, depending on the testing model (Every tick, Control points, Open prices only). If this parameter has not been specified, 0 is used (Every tick).

TestOptimization - enable/disable optimization. The values that can be taken are "true" or "false". If this parameter had not been specified, the "false" value is used.

TestDateEnable - enable/disable the "Use date" flag. The values that can be taken are "true" or "false". If this parameter had not been specified, the "false" value is used.

TestFromDate - the date, from which to start testing, appeared as YYYY.MM.DD. If this parameter has not been specified, this date is 1970.01.01.

TestToDate - the date, on which to finish testing, appeared as YYYY.MM.DD. If this parameter has not been specified, this date is 1970.01.01.

TestReport - the name of the test report file. The file will be created in the client terminal directory. A relative path can be specified, for example: testerMovingAverageReport". If the extension has not been specified in the file name, the ".htm" will be set automatically. If this parameter has not been specified, the test report will not be formed.

TestReplaceReport - enable/disable the repeated report file record. The values that can be taken are "true" or "false". If the "false" value is specified and a report file named in the same way exists already, the number in square brackets will be added to the file name. For example, "MovingAverageReport[1].htm". If this parameter had not been specified, the "false" value is used.

TestShutdownTerminal - enable/disable shutdown of the terminal after the testing has been finished. The values that can be taken are "true" or "false". If this parameter had not been specified, the "false" value is used. If the user has pressed the "Stop" button, the value of this parameter will be flushed to "false" since the control has been given to the user.

Example:

Code:
; start strategy tester
  TestExpert=Moving Average
  TestExpertParameters=ma0.set
  TestSymbol=EURUSD
  TestPeriod=H1
  TestModel=2
  TestOptimization=false
  TestDateEnable=true
  TestFromDate=1970.01.01
  TestToDate=2006.06.06
  TestReport=MovingAverageReport
  TestReplaceReport=false
  TestShutdownTerminal=true

History Center

Technical analysis is the market movements research made in order to forecast future price movements. the market is often analyzed using charts. So it is very important to have available historical data for all symbols and timeframes used. Historical data are constantly formed and stored on the server. Connecting to it, the client terminal downloads all necessary data. They will be then used for drawing of charts,testing and optimization of Expert Advisors. To control historical data, the terminal has a special window named "History Center". This window can be opened by executing the command "Tools — History Center" or by pressing F2.

After the terminal has been shut down, all accumulated historical data will be stored in the "History Center". Sizes of files containing historical quotes do not exceed values defined in settings. If the amount of historical data accumulated exceeeds the value set in the field of " Max. bars in history:", the oldest bars will be deleted when storing. For each timeframe, a separate history file is formed named as SSSSSSPP.hst (where SSSSSS - symbol name, PP - timeframe in minutes) and saved in the /HISTORY. Later on, the saved data will be used to draw charts, as well as for testing of trading strategies.

In the "History Center" window, the available data can be changed. For this, it is necessary to select the desired symbol and timeframe in the left part of the window.The corresponding data will be loaded in form of a table. To add a record about a new bar, it is necessary to press the button of the same name, fill out all necessary fields in the new window and press "OK". After that, the new bar will appear in the history. One can modify the bar by selecting the corresponding record and pressing the "Modify" button. To delete a bar, it is necessary to select it and press the button of the same name.

Load of Historical Data
It is possible to load quotes for basic currency pairs starting with year 1999 from the historical data server. To do it, it is necessary to select the desired symbol and press "Download".

Quote:
Attention: The loaded data can differ from historical data stored on the broker's trade server.

Upon pressing the button, data of M1 timeframe will be loaded. Other timeframes will be automatically recalculated from М1. At that, the time of the downloaded data will be automatically recalculated according to the active account time zone.

When downloading historical data, it is recommended to control amount of bars in history and in charts.

Quote:
Attention: The deeper is the history used, the more PC resources are needed.

Quotes are weekly updated on the server of historical data. Further, at restarts, only updated quotes will be downloaded.

Exports and Imports of Historical Data
Historical data can be exported into files formatted as CSV, PRN and HTM. For this, it is necessary to select the desired symbol in the left part of the "History Center" window and press "Export". Then it is necessary to select on of three file formats and specify the path of location on the hard disk.

Historical data as CSV, PRN, TXT, HTM and HST can also be imported into terminal.

Historical data in the file can be represented as follows (any other separator can be used instead of space):

YYYY.MM.DD HH:MM O H L C V

YYYY-MM-DD HH:MM O H L C V

YYYY/MM/DD HH:MM O H L C V

DD.MM.YYYY HH:MM O H L C V

DD-MM-YYYY HH:MM O H L C V

DD/MM/YYYY HH:MM O H L C V

First of all, it is necessary to select a symbol and a timeframe, for which the import will be performed, in the left part of the "History Center" window. Then it is necessary to set up import parameters by pressing "Import":

Separator — data separator in the file to be imported. Comma, semicolon, space or tabulation character can be used as separators;

Skip columns — skip columns when importing. This can be helpful when the imported file contains more data types than necessary;

Skip lines — skip rows (lines) when importing;

Time shift — shift data by several hours in time;

Selected only — import only selected data. Data are selected by lines using "Ctrl" and "Shift";

Volumes — enable/disable importing of volumes.

After historical data have been imported, they can be used to show charts and test Expert Advisors.

Historical File Format (HST Files)
The database header is the first

Code:
struct HistoryHeader
{
  int               version;            // database version
  char              copyright[64];      // copyright info
  char              symbol[12];         // symbol name
  int               period;             // symbol timeframe
  int               digits;             // the amount of digits after decimal point in the symbol
  time_t            timesign;           // timesign of the database creation
  time_t            last_sync;          // the last synchronization time
  int               unused[13];         // to be used in future
};

then goes the bars array (single-byte justification)

Code:
#pragma pack(push,1)
//---- standard representation of the quote in the database
struct RateInfo
{
  time_t            ctm;                // current time in seconds
  double            open;
  double            low;
  double            high;
  double            close;
  double            vol;
};
#pragma pack(pop)

Export of Quotes

 Source data serving as a basis for the entire analytical work of the terminal user are those about security price changes. This information is provided by the brokerage company. Price data allow to draw symbol charts, research in financial markets, use various trading tactics, and make trade decisions. Quotes represent files with records in format of "SYMBOL, BID, ASK, DATE" (security symbol, bid price, ask price, date and time) and income in the terminal automatically as soon as connection to the server has been established.

 The terminal allows to export the current quotes to other programs in the real-time mode through "DDE" (Dynamic Data Exchange) protocol. This is a protocol of operational systems of MS Windows used for dynamic data exchange among various applications. Quotes are given through DDE only at incoming of new ticks (ADVISE mode), but not immediately on request (REQUEST mode) where the latest price is shown. N/A is shown on the first REQUEST, and after the new price has been income, quotes will appear.

 To activate the export of quotes from the terminal through DDE, one has to enable the "Enable DDE server" option in the terminal settings.
Quote:
Attention: History Data cannot be exported through DDE protocol. The current quotes are exported only when the client terminal is online.
DDE request formats and their possible results by the example of "DDE-sample.xls" file:
Code:
  BID   request:    = MT4|BID!USDCHF         result:  1.5773  
   ASK   request:    = MT4|ASK!USDCHF         result:  1.5778  
   HIGH  request:    = MT4|HIGH!USDCHF        result:  1.5801  
   LOW   request:    = MT4|LOW!USDCHF         result:  1.5741  
   TIME  request:    = MT4|TIME!USDCHF        result:  21.05.02 9:52  
   QUOTE request:    = MT4|QUOTE!USDCHF       result:  21.05.02 9:52 1.5773 1.5778 1.5776
Quote:
Attention: For data to be shown properly in MS Excel, one has to enable "Tools — Options... — Translation — Translation formula entry" menu option of MS Excel.


Global Variables
 
 Several experts can be launched in the client terminal at the same time. Sometimes, there is a need them to interchange with information. To provide possibility of prompt transfer of moderate amounts of information among experts, as well as organize conflict-free simultaneous working of several experts, there are global variables in the terminal. Unlike variables claimed at a global level in the expert source code and available only within the corresponding module, global variables exsist independently on experts. Their values are saved between terminal launches, unlike those of variables claimed at a global level (they are set at every expert launch and lost at expert remove). Global variables are available within four weeks since their last call from experts or manual modifying.
 
 There is a special window in terminal that manages global variables. It can be opened by execution of the "Tools — Global Variables" menu command or by pressing of F3. All global variables, their values and times of their last calls are listed in a table in this window. Using buttons located in the right part of the window, one can add a new global variable or delete an existing one. To change the name or value of a global variable, one has to double-click with the left mouse button on the corresponding cell of the table. The last call time will be changed automatically for this variable. 


Contract Specification
 
 This message window allows to view securities contract specifications. The main parameters are grouped in table with following fields:
 
 Spread — difference between Bid and Ask prices in points;
 
 Digits — the amount of digits after decimal point in the price representation;
 
 Stops level — minimum distance to the current price in points at which Stop Loss and Take Profit orders can be placed;
 
 Pendings are good till cancel — forced closing of pending orders at the end of a session. "Yes" means that pending orders will not be closed forcedly;
 
 Contract size — one-lot price in deposit currency;
 
 Tick price — the size of minimal price change in quote currency;
 
 Tick size — minimal symbol price change interval in points;
 
 Profit calculation mode — accepted profit calculation technique (Forex, CFD, Futures);
 
 Swap type — rollover calculation type (in points, in deposit currency, or in per cents);
 
 Swap long — rollover size for a long position;
 
 Swap short — rollover size for a short position;
 
 Margin calculation mode — accepted free margin calculation technique (Forex, CFD, Futures)
 
 Margin hedge — size of margin for hedged positions.
 
 The symbol specification window can be called by pressing of "Properties" button in the Market Watch window or the "Symbol properties" of the "Tester — Settings" window.

Languages Support
 
 The Client Terminal interface is multilingual. It can be represented in any available language. This feature makes the program not only easy-to-use, but also allows to reduce the amount of errors that occur during trading because of lack of understanding of foreign terms and notions. The list of all available languages can be found in the "View — Languages" menu. One can switch one's terminal to a language by selecting of this language in the list. The language can be changed actually only after restart of the terminal. 

Datacenter.ini

Administrators of dealing centers who would like to redirect the clients' traffic forcedly have a new opportunity now. The Data Center used by the client terminal can be strictly defined using the configuration file named datacenter.ini. This file must be placed in the /config directory of the client terminal.

If the "Data Center auto configuration" is flagged in the server settings, there will be an attempt to open and analyze the datacenter.ini file. If a data center for the server will be found in this file, the connection will be performed exactly to it. No more than one data center can be defined for one server.

The string format of the datacenter.ini:

[server name][delimiter][data center address]

where:

server name is the name of the corresponding .srv file;

delimiter is a comma or any quantity of spaces;

data center address is the IP address : port number.

Lines that start with a semicolon ( are considered to be a comment and are not processed.

Exemplary datacenter.ini file:

Code:
  ;----------------
  ;comment
  ;----------------
  MetaQuotes-demo1  192.168.1.4:443
  MetaQuotes-demo2, 192.168.1.4:444
  ;----------------
  MetaQuotes-demo3  192.168.1.5:445
  MetaQuotes-demo4, 192.168.1.5:446



Filed under: 其它 No Comments
8Feb/12

GoAccess,Apache/Nginx 日志统计分析软件

Posted by Nick Xu

什么是GoAccess?

GoAccess是运行在Unix系统终端,开源、实时分析Apache日志(其实也能够分析Nginx日志)并且能够动态呈现的软件。它为系统管理员提供了一个快速、有价值的HTTP统计,动态的可视化的服务器报告。

它是做什么以及如何工作?

从本质上讲,GoAccess的工作方式是,它将解析众所周知的Apache访问日志文件 access log,GoAccess通过解析日志收集的数据,将会显示它的控制台或某个终端。收集到的信息或生成的报告将显示给在视觉、交互窗口的系统管理员用户。报告包括:

  • 综合统计
  • 访客排行榜
  • 请求文件排行榜
  • 请求的静态文件排行榜,如图片、JS文件、CSS文件等
  • 访问来源
  • 404页面
  • 访问者的操作系统
  • 访问者的浏览器
  • 浏览器和蜘蛛(搜索爬虫)
  • 主机,反向DNS,IP位置
  • HTTP状态码
  • 推介网站
  • 搜索关键词
  • 显示时不同的配色方案
  • 无限制日志文件的大小(决定了GoAccess的打开速度)

日志文件格式

GoAccess可以解析Apache的日志格式,通用日志格式(CLF)和组合日志格式(XLF/ ELF),包括虚拟主机. 它也能够解析Nginx日志. (如果你按照Apache日志的格式配置了Nginx日志)

GoAccess主页 http://goaccess.prosoftcorp.com/

安装方法

1、安装GoAccess需要一些系统支持库

Ncurses 提供字符终端处理库,包括面板和菜单
GLib C语言的函数库(大于2.0的版本)
GeoIP 通过IP,定位他的经纬度,国家/地区,省市,甚至街道等位置信息。(如果不需要这个不是必须的,配置时候去掉这个选项 –enable-geoip)

CentOS下yum安装

yum install ncurses-devel
yum install glib2-devel

Ubuntu下apt安装

apt-get install libglib2.0-dev
apt-get install libncursesw5-dev

2、安装GoAccess

CentOS下安装

wget http://sourceforge.net/projects/goaccess/files/0.4.2/goaccess-0.4.2.tar.gz
tar zxvf goaccess-0.4.2.tar.gz
cd goaccess-0.4.2
./configure --enable-utf8
make
make install

Ubuntu下安装

apt-get install goaccess

GoAccess使用介绍

启动参数介绍:

  • -b 流量消耗统计,如果想要提高解析速度,不要开启这个选项。缺省值为不开启此选项
  • -f 日志文件路径。
  • -s HTTP 状态码报告,为了能够更好的解析日志,选项被默认关闭。
  • -e 指定IP地址的访问统计。
  • -a 通过代理的请求统计。

菜单介绍

  • F1 帮助菜单
  • F5 刷新
  • q 退出当前口窗、菜单、或是当前查看的选项
  • o 打开当前的选项、菜单
  • c 改变窗口配色(目前只有两种 默认和绿色)
  • SHIFT + TAB 从当前选定模块向后切换
  • RIGHT 打开当前选中模块,查看详细信息
  • s 通过日期排序,只会在访问请求模块起作用
  • S 通过点击次数排序,只会在访问请求模块起作用
  • / 查看详细信息的窗口进行搜索
  • n 通过/进行查找后,查找下个匹配的内容的位置,如果没有则在窗口底部显示“search hit BOTTOM”
  • t 在查看详细信息窗口,移动指针到最顶部
  • b 在查看详细信息窗口,移动指针到最底部

使用实例

  • 最简单、快速的使用方式
goaccess -f /data1/logs/blog.wuwangwo.net/access.log
  • 产生完整、全面统计数据的使用方式
goaccess -f /data1/logs/blog.wuwangwo.net/access.log -a -s -b
  • 通过管道的方式将更多access log 通过GoAccess解析
zcat access.log.*.gz | goaccess

或者

zcat -f access.log* | goaccess
  • 另外一种通过管道使用GoAccess解析日志的方式
sed -n '/05/Dec/2010/,$ p' access.log | goaccess -s -b
  • 低优先级运行GoAccess的方式
nice -n 19 goaccess -f access.log -s -a -b
  • 如果你只把GoAccess安装到了一台服务器上
ssh user@server 'cat /var/log/apache2/access.log' | goaccess -s -a -b

注意事项

  1. 每一个详细信息窗口,只显示300条数据
  2. GoAccess 通过管道解析日志时将禁用实时解析的功能

总结

GoAceess 不是很大的一款软件却有着强大的功能即使软件可能还有些BUG,软件的更多介绍信息请查看GoAccess的官方网站«./GoAccess»,当前版本的官方手册页面 Man Page

Filed under: Linux No Comments
8Feb/12

Nginx Block And Deny IP Address OR Network Subnets

Posted by Nick Xu

How do I block or deny access based on the host name or IP address of the client visiting website under nginx web server?

Nginx comes with a simple module called ngx_http_access_module to allow or deny access to IP address. The syntax is as follows:

deny IP;
deny subnet;
allow IP;
allow subnet;
# block all ips
deny    all;
# allow all ips
allow    all;
Note rules are checked in the order of their record to the first match.

How Do I Configure Nginx To Block IPs?

Edit nginx.conf file, enter (note my nginx path is set to /usr/local/nginx/, replace this according to your setup):
# cd /usr/local/nginx/conf/
# vi nginx.conf

Add the following line in http section:

## Block spammers and other unwanted visitors  ##
include blockips.conf;
Save and close the file. Finally, create blockips.conf in /usr/local/nginx/conf/, enter:
# vi blockips.conf

Append / add entries as follows:

deny 1.2.3.4;
deny 91.212.45.0/24;
deny 91.212.65.0/24;

Save and close the file. Test the config file, enter:
# /usr/local/nginx/sbin/nginx -t

Sample outputs:

the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok
configuration file /usr/local/nginx/conf/nginx.conf test is successful
Reload the new config, enter:
# /usr/local/nginx/sbin/nginx -s reload

How Do I Deny All and Allow Only Intranet/LAN IPs?

Edit config file as follows:

location / {
# block one workstation
deny    192.168.1.1;
# allow anyone in 192.168.1.0/24
allow   192.168.1.0/24;
# drop rest of the world
deny    all;
}
Granted access to network 192.168.1.0/24 with the exception of the address 192.168.1.1.

How Do I Customize HTTP 403 Forbidden Error Messages?

Create a file called error403.html in default document root, enter:
# cd /usr/local/nginx/html
# vi error403.html

<html>
<head><title>Error 403 - IP Address Blocked</title></head>
<body>
Your IP Address is blocked. If you this an error, please contact webmaster with your IP at webmaster@example.com
</body>
</html>

If SSI enabled, you can display the client IP easily from the html page itself:

Your IP Address is <!--#echo var="REMOTE_ADDR" --> blocked.
Save and close the file. Edit your nginx.conf file, enter:
# vi nginx.conf

# redirect server error pages to the static page
error_page   403  /error403.html;
location = /error403.html {
root   html;
}
Save and close the file. Reload nginx, enter:
# /usr/local/nginx/sbin/nginx -s reload

See also:
Nginx Shell Script To Block Spamhaus Lasso Drop Spam IP Address

Filed under: Linux No Comments
7Feb/12

[转]附近地点搜索初探,非geohash

Posted by Nick Xu

附近地点搜索,顾名思义,就是搜索用户附近有哪些地点。随着GPS和带有GPS功能的移动设备的普及, 附近地点搜索也变得炙手可热。不过在网上却很少有这方面的讨论。本文的方法并不算最好, 但足以应付一般的应用了。

本文中,数据库采用MySQL,语言采用python。理论上别的数据库和语言也没问题, 但我们要在经纬度上设置两个索引,所以如果你的数据库不支持索引,或者不支持在一个查询中使用两个索引, 那就只能想别的办法了。

球面最短距离公式

球面上任意两点之间的距离计算公式可以参考维基百科上的下述文章,这里就不再赘述了。

值得一提的是,维基百科推荐使用Haversine公式,理由是Great-circle distance公式用到了大量余弦函数, 而两点间距离很短时(比如地球表面上相距几百米的两点),余弦函数会得出0.999...的结果, 会导致较大的舍入误差。而Haversine公式采用了正弦函数,即使距离很小,也能保持足够的有效数字。 以前采用三角函数表计算时的确会有这个问题,但经过实际验证,采用计算机来计算时,两个公式的区别不大。 稳妥起见,这里还是采用Haversine公式。

distance-haversin-distance.png

其中

distance-haversin.png
  • R为地球半径,可取平均值 6371km;
  • φ1, φ2 表示两点的纬度;
  • Δλ 表示两点经度的差值。

距离计算函数

下面就是计算球面间两点(lat0, lng)-(lat1, lng1)之间距离的函数。

from math import sin, asin, cos, radians, fabs, sqrt

EARTH_RADIUS=6371           # 地球平均半径,6371km

def hav(theta):
    s = sin(theta / 2)
    return s * s

def get_distance_hav(lat0, lng0, lat1, lng1):
    "用haversine公式计算球面两点间的距离。"
    # 经纬度转换成弧度
    lat0 = radians(lat0)
    lat1 = radians(lat1)
    lng0 = radians(lng0)
    lng1 = radians(lng1)

    dlng = fabs(lng0 - lng1)
    dlat = fabs(lat0 - lat1)
    h = hav(dlat) + cos(lat0) * cos(lat1) * hav(dlng)
    distance = 2 * EARTH_RADIUS * asin(sqrt(h))

    return distance

范围搜索算法

在庞大的地理数据库中搜索地点,索引是很重要的。但是,我们的需求是搜索附近地点, 例如,坐标(39.91, 116.37)附近500米内有什么地点?搜索条件是地点坐标与当前坐标之间的距离, 显然是无法应用索引的。

那么换个思路:首先算出“给定坐标附近500米”这个范围的坐标范围。 虽然它是个圆,但我们可以先求出该圆的外接正方形,然后拿正方形的经纬度范围去搜索数据库。

distance-map.png

如图,红色部分为要求的搜索范围,绿色部分为实际搜索范围。

先来求东西两侧的的范围边界。在haversin公式中令φ1 = φ2,可得

distance-lng.png

写成python代码就是

dlng = 2 * asin(sin(distance / (2 * EARTH_RADIUS)) / cos(lat))
dlng = degrees(dlng)        # 弧度转换成角度

然后求南北两侧的范围边界,在haversin公式中令 Δλ = 0,可得

distance-lat.png

写成python代码就是

dlat = distance / EARTH_RADIUS
dlng = degrees(dlat)     # 弧度转换成角度

这样,根据当前点坐标,我们可以得出搜索范围为

left-top    : (lat + dlat, lng - dlng)
right-top   : (lat + dlat, lng + dlng)
left-bottom : (lat - dlat, lng - dlng)
right-bottom: (lat - dlat, lng + dlng)

然后利用这个范围构造SQL语句,即可实现范围查询:

SELECT * FROM place WHERE lat > lat1 AND lat < lat2 AND lng > lng1 AND lng < lng2;

在lat和lng列上建立索引,能从一定程度上提高范围查询的效率。

不过,这样查询到的地点是正方形范围内的地点,一些结果与当前点的距离可能会超出给定的距离。 如果要求严格,可以遍历结果并计算与当前点之间的距离,并过滤掉不符合要求的结果。

总结

附近地点搜索条件是距离,而数据库中一般只保存地点的经纬度,因此无法直接查询。 本文将距离转化成经纬度范围,利用经纬度上的索引,提高查询效率。

Filed under: Python No Comments
7Feb/12

[转]禁止爬虫访问技巧

Posted by Nick Xu

有时候,一些秘密的入口,一些被爬虫爬过便有暴露风险的目录,我们必须采用一切手段来保卫。禁止一切爬虫进入,尽管称不上完全保密,但基本上可以把秘密的传播变得可控。

1.robots.txt

在想要保密的那个目录或者入口的根目录创建robots.txt,然后在里面加入以下内容,则听话的爬虫如Google爬虫在抓到你的网站的时候就会悄然离去。

User-Agent: *
Disallow: /

2.htaccess

由于并不是所有搜索引擎都像Google那样Do No Evil,所以并不是所有爬虫都像Google爬虫那样听话的遵循Robots.txt里的描述,这个时候我们需要用到htaccess来强制屏蔽那些恶 心的坏爬虫们的访问,这种屏蔽是直接的屏蔽。我们可以在要保密的那个目录或者入口的根目录的.htaccess文件中加入以下内容:

RewriteCond %{HTTP_USER_AGENT} ^BlackWidow [OR]
RewriteCond %{HTTP_USER_AGENT} ^Bot\ mailto:craftbot@yahoo.com [OR]
RewriteCond %{HTTP_USER_AGENT} ^ChinaClaw [OR]
RewriteCond %{HTTP_USER_AGENT} ^Custo [OR]
RewriteCond %{HTTP_USER_AGENT} ^DISCo [OR]
RewriteCond %{HTTP_USER_AGENT} ^Download\ Demon [OR]
RewriteCond %{HTTP_USER_AGENT} ^eCatch [OR]
RewriteCond %{HTTP_USER_AGENT} ^EirGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailSiphon [OR]
RewriteCond %{HTTP_USER_AGENT} ^EmailWolf [OR]
RewriteCond %{HTTP_USER_AGENT} ^Express\ WebPictures [OR]
RewriteCond %{HTTP_USER_AGENT} ^ExtractorPro [OR]
RewriteCond %{HTTP_USER_AGENT} ^EyeNetIE [OR]
RewriteCond %{HTTP_USER_AGENT} ^FlashGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetRight [OR]
RewriteCond %{HTTP_USER_AGENT} ^GetWeb! [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go!Zilla [OR]
RewriteCond %{HTTP_USER_AGENT} ^Go-Ahead-Got-It [OR]
RewriteCond %{HTTP_USER_AGENT} ^GrabNet [OR]
RewriteCond %{HTTP_USER_AGENT} ^Grafula [OR]
RewriteCond %{HTTP_USER_AGENT} ^HMView [OR]
RewriteCond %{HTTP_USER_AGENT} HTTrack [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Stripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^Image\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} Indy\ Library [NC,OR]
RewriteCond %{HTTP_USER_AGENT} ^InterGET [OR]
RewriteCond %{HTTP_USER_AGENT} ^Internet\ Ninja [OR]
RewriteCond %{HTTP_USER_AGENT} ^JetCar [OR]
RewriteCond %{HTTP_USER_AGENT} ^JOC\ Web\ Spider [OR]
RewriteCond %{HTTP_USER_AGENT} ^larbin [OR]
RewriteCond %{HTTP_USER_AGENT} ^LeechFTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mass\ Downloader [OR]
RewriteCond %{HTTP_USER_AGENT} ^MIDown\ tool [OR]
RewriteCond %{HTTP_USER_AGENT} ^Mister\ PiX [OR]
RewriteCond %{HTTP_USER_AGENT} ^Navroad [OR]
RewriteCond %{HTTP_USER_AGENT} ^NearSite [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetAnts [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Net\ Vampire [OR]
RewriteCond %{HTTP_USER_AGENT} ^NetZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Octopus [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Explorer [OR]
RewriteCond %{HTTP_USER_AGENT} ^Offline\ Navigator [OR]
RewriteCond %{HTTP_USER_AGENT} ^PageGrabber [OR]
RewriteCond %{HTTP_USER_AGENT} ^Papa\ Foto [OR]
RewriteCond %{HTTP_USER_AGENT} ^pavuk [OR]
RewriteCond %{HTTP_USER_AGENT} ^pcBrowser [OR]
RewriteCond %{HTTP_USER_AGENT} ^RealDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^ReGet [OR]
RewriteCond %{HTTP_USER_AGENT} ^SiteSnagger [OR]
RewriteCond %{HTTP_USER_AGENT} ^SmartDownload [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperBot [OR]
RewriteCond %{HTTP_USER_AGENT} ^SuperHTTP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Surfbot [OR]
RewriteCond %{HTTP_USER_AGENT} ^tAkeOut [OR]
RewriteCond %{HTTP_USER_AGENT} ^Teleport\ Pro [OR]
RewriteCond %{HTTP_USER_AGENT} ^VoidEYE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Image\ Collector [OR]
RewriteCond %{HTTP_USER_AGENT} ^Web\ Sucker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebAuto [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebCopier [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebFetch [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebGo\ IS [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebLeacher [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebReaper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebSauger [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ eXtractor [OR]
RewriteCond %{HTTP_USER_AGENT} ^Website\ Quester [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebStripper [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebWhacker [OR]
RewriteCond %{HTTP_USER_AGENT} ^WebZIP [OR]
RewriteCond %{HTTP_USER_AGENT} ^Wget [OR]
RewriteCond %{HTTP_USER_AGENT} ^Widow [OR]
RewriteCond %{HTTP_USER_AGENT} ^WWWOFFLE [OR]
RewriteCond %{HTTP_USER_AGENT} ^Xaldon\ WebSpider [OR]
RewriteCond %{HTTP_USER_AGENT} ^Zeus [or]
RewriteCond %{HTTP_USER_AGENT} ^BaiduSpider
RewriteRule ^.* – [F,L]

 

nginx 用 if 和 $http_user_agent。

如:

  1. location / {
  2.     root   /home/www/;
  3.     if ($http_user_agent ~* "qihoobot") {
  4.         return 403;
  5.     }
  6.     #...
  7. }
location / {
	root   /home/www/;

	if ($http_user_agent ~* "qihoobot") {
		return 403;
	}
	#...
}

然后重启 nginx

Filed under: Linux No Comments
7Feb/12

转Linux流量监控工具 – iftop (最全面的iftop教程)

Posted by Nick Xu

在类Unix系统中可以使用top查看系统资源、进程、内存占用等信息。查看网络状态可以使用netstat、nmap等工具。若要查看实时的网络流量,监控TCP/IP连接等,则可以使用iftop

一、iftop是什么?

iftop是类似于top的实时流量监控工具。

官方网站:http://www.ex-parrot.com/~pdw/iftop/

二、iftop有什么用?

iftop可以用来监控网卡的实时流量(可以指定网段)、反向解析IP、显示端口信息等,详细的将会在后面的使用参数中说明。

三、安装iftop

安装方法1、编译安装

如果采用编译安装可以到iftop官网下载最新的源码包。

安装前需要已经安装好基本的编译所需的环境,比如make、gcc、autoconf等。安装iftop还需要安装libpcap和libcurses。

CentOS上安装所需依赖包:

yum install flex byacc  libpcap ncurses ncurses-devel libpcap-devel

Debian上安装所需依赖包:

apt-get install flex byacc  libpcap0.8 libncurses5

下载iftop

wget http://www.ex-parrot.com/pdw/iftop/download/iftop-0.17.tar.gz

tar zxvf iftop-0.17.tar.gz

cd iftop-0.17

./configure

make && make install

安装方法2:(懒人办法,最简单)

直接省略上面的步骤

CentOS系统:

yum install flex byacc  libpcap ncurses ncurses-devel

wget ftp://fr2.rpmfind.net/linux/dag/redhat/el5/en/i386/dag/RPMS/iftop-0.17-1.el5.rf.i386.rpm

rpm -ivh iftop-0.17-1.el5.rf.i386.rpm

Debian系统 运行:apt-get install iftop

四、运行iftop

直接运行: iftop

效果如下图:

五、相关参数及说明

1、iftop界面相关说明

界面上面显示的是类似刻度尺的刻度范围,为显示流量图形的长条作标尺用的。

中间的<= =>这两个左右箭头,表示的是流量的方向。

TX:发送流量
RX:接收流量
TOTAL:总流量
Cumm:运行iftop到目前时间的总流量
peak:流量峰值
rates:分别表示过去 2s 10s 40s 的平均流量

2、iftop相关参数

常用的参数

-i设定监测的网卡,如:# iftop -i eth1

-B 以bytes为单位显示流量(默认是bits),如:# iftop -B

-n使host信息默认直接都显示IP,如:# iftop -n

-N使端口信息默认直接都显示端口号,如: # iftop -N

-F显示特定网段的进出流量,如# iftop -F 10.10.1.0/24或# iftop -F 10.10.1.0/255.255.255.0

-h(display this message),帮助,显示参数信息

-p使用这个参数后,中间的列表显示的本地主机信息,出现了本机以外的IP信息;

-b使流量图形条默认就显示;

-f这个暂时还不太会用,过滤计算包用的;

-P使host信息及端口信息默认就都显示;

-m设置界面最上边的刻度的最大值,刻度分五个大段显示,例:# iftop -m 100M

进入iftop画面后的一些操作命令(注意大小写)

按h切换是否显示帮助;

按n切换显示本机的IP或主机名;

按s切换是否显示本机的host信息;

按d切换是否显示远端目标主机的host信息;

按t切换显示格式为2行/1行/只显示发送流量/只显示接收流量;

按N切换显示端口号或端口服务名称;

按S切换是否显示本机的端口信息;

按D切换是否显示远端目标主机的端口信息;

按p切换是否显示端口信息;

按P切换暂停/继续显示;

按b切换是否显示平均流量图形条;

按B切换计算2秒或10秒或40秒内的平均流量;

按T切换是否显示每个连接的总流量;

按l打开屏幕过滤功能,输入要过滤的字符,比如ip,按回车后,屏幕就只显示这个IP相关的流量信息;

按L切换显示画面上边的刻度;刻度不同,流量图形条会有变化;

按j或按k可以向上或向下滚动屏幕显示的连接记录;

按1或2或3可以根据右侧显示的三列流量数据进行排序;

按<根据左边的本机名或IP排序;

按>根据远端目标主机的主机名或IP排序;

按o切换是否固定只显示当前的连接;

按f可以编辑过滤代码,这是翻译过来的说法,我还没用过这个!

按!可以使用shell命令,这个没用过!没搞明白啥命令在这好用呢!

按q退出监控。

六、常见问题

1、make: yacc: Command not found
make: *** [grammar.c] Error 127

解决方法:apt-get install byacc   /   yum install byacc

2、configure: error: Curses! Foiled again!
(Can't find a curses library supporting mvchgat.)
Consider installing ncurses.

解决方法:apt-get install libncurses5-dev  /    yum  install ncurses-devel

Filed under: Linux No Comments
5Feb/12

Nginx的防盗链和限速处理

Posted by Nick Xu

防盗链处理

1 简单的通过referer判断

location ~ .*\.(gif|jpg|jpeg|png|bmp|wma|mp3|swf)$ {
      valid_referers none blocked server_names *.163.com 163.com baidu.com;
      if ($invalid_referer) {return 403;}
    expires      30d;
}

2 NginxHttpAccessKeyModule 插件

这个需要安装额外的模块,下载模块源码,然后按照说明文档,将下载来的源码中”$HTTP_ACCESSKEY_MODULE” 替换成 “ngx_http_accesskey_module“,再

  1. ./configure –add-module=path/to/nginx-accesskey

完了以后再重新编译一次nginx,然后就可以在配置文件中使了。

location /download {
  accesskey             on;
  accesskey_hashmethod  md5;
  accesskey_arg         "key";
  accesskey_signature   "mypass$remote_addr";
}

这样一个下载的文件就需要 http://example.com/download/file.zip?key=09093abeac094 这样的形式进行下载
其中 key的值为 mypass和下载客户端ip 进行md5运算获得。比如在 php 中可以向如下方式输出针对客户ip的下载链接地址

  1. <?
  2. $ipkeyvalue = md5("mypass".$_SERVER['REMOTE_ADDR']);
  3. $out_keyfile_link="<a href=http://example.com/download/testfile.rar?key=".$ipkeyvalue.">防盗链文件下载</a><br />";
  4. echo $out_keyfile_link;
  5. ?>

这样当一个用户将访问地址拷贝给别人时,因为访问ip不同,就造成 md5(“mypass”.ipaddr) 值不同,达到防盗链目的。

限速处理

限速使用 limit_zone, limit_conn 以及 limit_rate 进行配置
首先在 http 段配置一个 limit_zone,然后在需要的地方使用 limit_conn 和 limit_rate 进行限速设置,如下一个简单的例子

http {
  limit_zone   one  $binary_remote_addr  10m;
  server {
    location /files/ {
      limit_conn   one  1;
      limit_rate 20k;
    }
  }
}

说明:
limit_zone,是针对每个IP定义一个存储session状态的容器。这个示例中定义了一个名叫one的10m大小的容器,这个名字会在后面的limit_conn中使用。

limit_conn one 1;
限制在one中记录状态的每个IP只能发起一个并发连接。

limit_rate 20k;
对每个连接限速20k. 注意这里是对连接限速,而不是对IP限速。如果一个IP允许三个并发连接,那么这个IP就是限速为limit_rate×3,在设置的时候要根据自己的需要做设置调整,要不然会达不到自己希望的目的。

Filed under: Linux No Comments
   
site
site