↓ 跳过正文
  1. Posts/

docker network 与 本地 network 网段冲突

·537 字·2 分钟

起因:
#

公司部署在 hk 的爬虫服务器突然挂掉了。后来发现只是在深圳办公区无法访问。排查后发现原因是 docker 的网络(包括 docker network 的 subnet 或者是某个容器的 ip)与该 host 在内网的 ip 段相同,导致冲突。

排查过程:
#

有两个方面需要排查。一个是 docker 服务启动时的默认网络。

默认网络使用 bridge 桥接模式,是容器与宿主机进行通讯的默认办法。

修改默认网段可以参考 http://blog.51cto.com/wsxxsl/2060761

除此之外,还需要注意 docker 创建的 network 的网段。

使用 docker network ls 命令查看当前的网络。

然后可以使用 docker inspect 查看每个 network 的详细信息。

也可以直接使用 ip addr 来查看各种奇怪的虚拟网卡的 ip,是否有前两位的地址和 host 的 ip 地址相同。

解决办法:
#

本想在 docker-compose up 时指定默认网络的 subnet。

结果发现好像并不支持?version 1.10.0 error on gateway spec

Was there any discussion on that? I do need to customize the network, because my company uses the 172.16.0.0/16 address range at some segments and Docker will simply clash with that by default, so every single Docker server in the whole company needs a forced network setting.

Now while upgrading my dev environment to Docker 1.13 it took me hours to stumble into this Github issue, because the removal of those options was completely undocumented.

So please, if I am working on a network which requires a custom docker subnet, how am I supposed to use Docker Compose and Docker Swarm?

最后用了个比较间接的办法。

先手动创建一个 docker network,然后再在 docker-compose 的配置文件中指定。

1docker network create --subnet=172.25.0.0/16 spider-network
 1version: '3.6'
 2services:
 3  django:
 4    image: "registry.sensetime.com/spider/sensespider:v1.0"
 5    volumes:
 6      - spiderdata:/data
 7    privileged: true
 8    networks:
 9      - spider-network
10    ports:
11      - "8080:8080"
12    working_dir: /home/renkuanze/workspace/sensespider
13    entrypoint: bash run.sh
14    tty: true
15    container_name: django
16  splash:
17    image: "scrapinghub/splash"
18    container_name: splash
19    network_mode: "service:django"
20  react:
21    image: "registry.sensetime.com/spider/sensespider-react:v1.0"
22    working_dir: /home/node/sensespider-react
23    entrypoint: npm start
24    container_name: node
25    network_mode: "service:django"
26volumes:
27  spiderdata:
28    driver: local-persist
29    driver_opts:
30      mountpoint: /data/spider_data
31
32networks:
33  spider-network:
34    external: true

相关文章

How to use Scrapy with Django Application(转自medium)

·2519 字·6 分钟
在 medium 上看到一篇很赞的文章……无奈关键部分一律无法加载出来,挂了梯子也不行,很心塞。刚刚突然发现加载出来了,以防之后再次无法访问,所以搬运过来。 There are couple of articles on how to integrate Scrapy into a Django Application (or vice versa?). But most of them don’t cover a full complete example that includes triggering spiders from Django views. Since this is a web application, that must be our main goal.

lua学习笔记

·2216 字·5 分钟
lua 是一门轻量级的脚本语言,好像比较适合写游戏?我在 太阳神三国杀 中见过很多 lua 脚本。由于 splash 的渲染脚本需要用 lua 来写,因此来学习一波。 直接上语法,看到了 python 和 pascal 的影子 orz。 详细代码 1-- Two dashes start a one-line comment. 2 3--[[ 4 Adding two ['s and ]'s makes it a 5 multi-line comment. 6--]] 7 8---------------------------------------------------- 9-- 1. Variables and flow control. 10---------------------------------------------------- 11 12num = 42 -- All numbers are doubles. 13-- Don't freak out, 64-bit doubles have 52 bits for 14-- storing exact int values; machine precision is 15-- not a problem for ints that need < 52 bits. 16 17s = 'walternate' -- Immutable strings like Python. 18t = "double-quotes are also fine" 19u = [[ Double brackets 20 start and end 21 multi-line strings.]] 22t = nil -- Undefines t; Lua has garbage collection. 23 24-- Blocks are denoted with keywords like do/end: 25while num < 50 do 26 num = num + 1 -- No ++ or += type operators. 27end 28 29-- If clauses: 30if num > 40 then 31 print('over 40') 32elseif s ~= 'walternate' then -- ~= is not equals. 33 -- Equality check is == like Python; ok for strs. 34 io.write('not over 40\n') -- Defaults to stdout. 35else 36 -- Variables are global by default. 37 thisIsGlobal = 5 -- Camel case is common. 38 39 -- How to make a variable local: 40 local line = io.read() -- Reads next stdin line. 41 42 -- String concatenation uses the .. operator: 43 print('Winter is coming, ' .. line) 44end 45 46-- Undefined variables return nil. 47-- This is not an error: 48foo = anUnknownVariable -- Now foo = nil. 49 50aBoolValue = false 51 52-- Only nil and false are falsy; 0 and '' are true! 53if not aBoolValue then print('twas false') end 54 55-- 'or' and 'and' are short-circuited. 56-- This is similar to the a?b:c operator in C/js: 57ans = aBoolValue and 'yes' or 'no' --> 'no' 58 59karlSum = 0 60for i = 1, 100 do -- The range includes both ends. 61 karlSum = karlSum + i 62end 63 64-- Use "100, 1, -1" as the range to count down: 65fredSum = 0 66for j = 100, 1, -1 do fredSum = fredSum + j end 67 68-- In general, the range is begin, end[, step]. 69 70-- Another loop construct: 71repeat 72 print('the way of the future') 73 num = num - 1 74until num == 0 75 76 77---------------------------------------------------- 78-- 2. Functions. 79---------------------------------------------------- 80 81function fib(n) 82 if n < 2 then return 1 end 83 return fib(n - 2) + fib(n - 1) 84end 85 86-- Closures and anonymous functions are ok: 87function adder(x) 88 -- The returned function is created when adder is 89 -- called, and remembers the value of x: 90 return function (y) return x + y end 91end 92a1 = adder(9) 93a2 = adder(36) 94print(a1(16)) --> 25 95print(a2(64)) --> 100 96 97-- Returns, func calls, and assignments all work 98-- with lists that may be mismatched in length. 99-- Unmatched receivers are nil; 100-- unmatched senders are discarded. 101 102x, y, z = 1, 2, 3, 4 103-- Now x = 1, y = 2, z = 3, and 4 is thrown away. 104 105function bar(a, b, c) 106 print(a, b, c) 107 return 4, 8, 15, 16, 23, 42 108end 109 110x, y = bar('zaphod') --> prints "zaphod nil nil" 111-- Now x = 4, y = 8, values 15..42 are discarded. 112 113-- Functions are first-class, may be local/global. 114-- These are the same: 115function f(x) return x * x end 116f = function (x) return x * x end 117 118-- And so are these: 119local function g(x) return math.sin(x) end 120local g; g = function (x) return math.sin(x) end 121-- the 'local g' decl makes g-self-references ok. 122 123-- Trig funcs work in radians, by the way. 124 125-- Calls with one string param don't need parens: 126print 'hello' -- Works fine. 127 128 129---------------------------------------------------- 130-- 3. Tables. 131---------------------------------------------------- 132 133-- Tables = Lua's only compound data structure; 134-- they are associative arrays. 135-- Similar to php arrays or js objects, they are 136-- hash-lookup dicts that can also be used as lists. 137 138-- Using tables as dictionaries / maps: 139 140-- Dict literals have string keys by default: 141t = {key1 = 'value1', key2 = false} 142 143-- String keys can use js-like dot notation: 144print(t.key1) -- Prints 'value1'. 145t.newKey = {} -- Adds a new key/value pair. 146t.key2 = nil -- Removes key2 from the table. 147 148-- Literal notation for any (non-nil) value as key: 149u = {['@!#'] = 'qbert', [{}] = 1729, [6.28] = 'tau'} 150print(u[6.28]) -- prints "tau" 151 152-- Key matching is basically by value for numbers 153-- and strings, but by identity for tables. 154a = u['@!#'] -- Now a = 'qbert'. 155b = u[{}] -- We might expect 1729, but it's nil: 156-- b = nil since the lookup fails. It fails 157-- because the key we used is not the same object 158-- as the one used to store the original value. So 159-- strings & numbers are more portable keys. 160 161-- A one-table-param function call needs no parens: 162function h(x) print(x.key1) end 163h{key1 = 'Sonmi~451'} -- Prints 'Sonmi~451'. 164 165for key, val in pairs(u) do -- Table iteration. 166 print(key, val) 167end 168 169-- _G is a special table of all globals. 170print(_G['_G'] == _G) -- Prints 'true'. 171 172-- Using tables as lists / arrays: 173 174-- List literals implicitly set up int keys: 175v = {'value1', 'value2', 1.21, 'gigawatts'} 176for i = 1, #v do -- #v is the size of v for lists. 177 print(v[i]) -- Indices start at 1 !! SO CRAZY! 178end 179-- A 'list' is not a real type. v is just a table 180-- with consecutive integer keys, treated as a list. 181 182---------------------------------------------------- 183-- 3.1 Metatables and metamethods. 184---------------------------------------------------- 185 186-- A table can have a metatable that gives the table 187-- operator-overloadish behavior. Later we'll see 188-- how metatables support js-prototypey behavior. 189 190f1 = {a = 1, b = 2} -- Represents the fraction a/b. 191f2 = {a = 2, b = 3} 192 193-- This would fail: 194-- s = f1 + f2 195 196metafraction = {} 197function metafraction.__add(f1, f2) 198 sum = {} 199 sum.b = f1.b * f2.b 200 sum.a = f1.a * f2.b + f2.a * f1.b 201 return sum 202end 203 204setmetatable(f1, metafraction) 205setmetatable(f2, metafraction) 206 207s = f1 + f2 -- call __add(f1, f2) on f1's metatable 208 209-- f1, f2 have no key for their metatable, unlike 210-- prototypes in js, so you must retrieve it as in 211-- getmetatable(f1). The metatable is a normal table 212-- with keys that Lua knows about, like __add. 213 214-- But the next line fails since s has no metatable: 215-- t = s + s 216-- Class-like patterns given below would fix this. 217 218-- An __index on a metatable overloads dot lookups: 219defaultFavs = {animal = 'gru', food = 'donuts'} 220myFavs = {food = 'pizza'} 221setmetatable(myFavs, {__index = defaultFavs}) 222eatenBy = myFavs.animal -- works! thanks, metatable 223 224-- Direct table lookups that fail will retry using 225-- the metatable's __index value, and this recurses. 226 227-- An __index value can also be a function(tbl, key) 228-- for more customized lookups. 229 230-- Values of __index,add, .. are called metamethods. 231-- Full list. Here a is a table with the metamethod. 232 233-- __add(a, b) for a + b 234-- __sub(a, b) for a - b 235-- __mul(a, b) for a * b 236-- __div(a, b) for a / b 237-- __mod(a, b) for a % b 238-- __pow(a, b) for a ^ b 239-- __unm(a) for -a 240-- __concat(a, b) for a .. b 241-- __len(a) for #a 242-- __eq(a, b) for a == b 243-- __lt(a, b) for a < b 244-- __le(a, b) for a <= b 245-- __index(a, b) <fn or a table> for a.b 246-- __newindex(a, b, c) for a.b = c 247-- __call(a, ...) for a(...) 248 249---------------------------------------------------- 250-- 3.2 Class-like tables and inheritance. 251---------------------------------------------------- 252 253-- Classes aren't built in; there are different ways 254-- to make them using tables and metatables. 255 256-- Explanation for this example is below it. 257 258Dog = {} -- 1. 259 260function Dog:new() -- 2. 261 newObj = {sound = 'woof'} -- 3. 262 self.__index = self -- 4. 263 return setmetatable(newObj, self) -- 5. 264end 265 266function Dog:makeSound() -- 6. 267 print('I say ' .. self.sound) 268end 269 270mrDog = Dog:new() -- 7. 271mrDog:makeSound() -- 'I say woof' -- 8. 272 273-- 1. Dog acts like a class; it's really a table. 274-- 2. function tablename:fn(...) is the same as 275-- function tablename.fn(self, ...) 276-- The : just adds a first arg called self. 277-- Read 7 & 8 below for how self gets its value. 278-- 3. newObj will be an instance of class Dog. 279-- 4. self = the class being instantiated. Often 280-- self = Dog, but inheritance can change it. 281-- newObj gets self's functions when we set both 282-- newObj's metatable and self's __index to self. 283-- 5. Reminder: setmetatable returns its first arg. 284-- 6. The : works as in 2, but this time we expect 285-- self to be an instance instead of a class. 286-- 7. Same as Dog.new(Dog), so self = Dog in new(). 287-- 8. Same as mrDog.makeSound(mrDog); self = mrDog. 288 289---------------------------------------------------- 290 291-- Inheritance example: 292 293LoudDog = Dog:new() -- 1. 294 295function LoudDog:makeSound() 296 s = self.sound .. ' ' -- 2. 297 print(s .. s .. s) 298end 299 300seymour = LoudDog:new() -- 3. 301seymour:makeSound() -- 'woof woof woof' -- 4. 302 303-- 1. LoudDog gets Dog's methods and variables. 304-- 2. self has a 'sound' key from new(), see 3. 305-- 3. Same as LoudDog.new(LoudDog), and converted to 306-- Dog.new(LoudDog) as LoudDog has no 'new' key, 307-- but does have __index = Dog on its metatable. 308-- Result: seymour's metatable is LoudDog, and 309-- LoudDog.__index = LoudDog. So seymour.key will 310-- = seymour.key, LoudDog.key, Dog.key, whichever 311-- table is the first with the given key. 312-- 4. The 'makeSound' key is found in LoudDog; this 313-- is the same as LoudDog.makeSound(seymour). 314 315-- If needed, a subclass's new() is like the base's: 316function LoudDog:new() 317 newObj = {} 318 -- set up newObj 319 self.__index = self 320 return setmetatable(newObj, self) 321end 322 323---------------------------------------------------- 324-- 4. Modules. 325---------------------------------------------------- 326 327 328--[[ I'm commenting out this section so the rest of 329-- this script remains runnable. 330-- Suppose the file mod.lua looks like this: 331local M = {} 332 333local function sayMyName() 334 print('Hrunkner') 335end 336 337function M.sayHello() 338 print('Why hello there') 339 sayMyName() 340end 341 342return M 343 344-- Another file can use mod.lua's functionality: 345local mod = require('mod') -- Run the file mod.lua. 346 347-- require is the standard way to include modules. 348-- require acts like: (if not cached; see below) 349local mod = (function () 350 <contents of mod.lua> 351end)() 352-- It's like mod.lua is a function body, so that 353-- locals inside mod.lua are invisible outside it. 354 355-- This works because mod here = M in mod.lua: 356mod.sayHello() -- Says hello to Hrunkner. 357 358-- This is wrong; sayMyName only exists in mod.lua: 359mod.sayMyName() -- error 360 361-- require's return values are cached so a file is 362-- run at most once, even when require'd many times. 363 364-- Suppose mod2.lua contains "print('Hi!')". 365local a = require('mod2') -- Prints Hi! 366local b = require('mod2') -- Doesn't print; a=b. 367 368-- dofile is like require without caching: 369dofile('mod2.lua') --> Hi! 370dofile('mod2.lua') --> Hi! (runs it again) 371 372-- loadfile loads a lua file but doesn't run it yet. 373f = loadfile('mod2.lua') -- Call f() to run it. 374 375-- loadstring is loadfile for strings. 376g = loadstring('print(343)') -- Returns a function. 377g() -- Prints out 343; nothing printed before now. 378 379--]] 380 381---------------------------------------------------- 382-- 5. References. 383---------------------------------------------------- 384 385--[[ 386 387I was excited to learn Lua so I could make games 388with the Löve 2D game engine. That's the why. 389 390I started with BlackBulletIV's Lua for programmers. 391Next I read the official Programming in Lua book. 392That's the how. 393 394It might be helpful to check out the Lua short 395reference on lua-users.org. 396 397The main topics not covered are standard libraries: 398 * string library 399 * table library 400 * math library 401 * io library 402 * os library 403 404By the way, this entire file is valid Lua; save it 405as learn.lua and run it with "lua learn.lua" ! 406 407This was first written for tylerneylon.com. It's 408also available as a github gist. Tutorials for other 409languages, in the same style as this one, are here: 410 411http://learnxinyminutes.com/ 412 413Have fun with Lua! 414 415--]]

golang 学习笔记

·358 字·1 分钟
先放资料,可能比较侧重于go在系统调用方面的内容. 这里不会记录详细的go的语法,只会记录学习的过程,踩到的坑,以及其他我认为值得记录的内容. go的switch语句终于是人类思维的语句了…匹配中了不需要加break..

爬虫学习笔记

·1182 字·3 分钟
再次迫于生计。。。 参考了面向新人的 Python 爬虫学习资料 大致的学习路线为: 一: 简单的定向脚本爬虫( request — bs4 — re ) 二: 大型框架式爬虫( Scrapy 框架为主) 三:浏览器模拟爬虫 ( Mechanize 模拟 和 Selenium 模拟) 有 Python 基础和一点 html 基础的话,貌似上手是 0 难度的。