Crawler
======================================================================

2010-02-02
----------------------------------------------------------------------

Chimera Search:
  Hardware: Sun Fire X4100
    CPU: AMD Opteron 254 (1 cores, 2.8GHz) x 2
    Memory: 16GB
    HDD: SEAGATE ST973401LSUN72G (SAS, 72GB, 10krpm, 2.5inch) x 2
  Software:
    OS: CentOS 5 (x86-64)
    Kernel: Linux 2.6.18-53.el5PAE
    Filesystem: ext3, RAID1 (2 disks) by LSI Fusion MPT SAS
    Packages:
      osstech-chimera-0.10.8-8.el5
      osstech-hyperestraier-1.4.13-23.0.el5
      osstech-qdbm-1.8.77-7.0.el5

SMB server:
  Hardware: HP ProLiant DL385 G2
    CPU: AMD Opteron 2216 HE (2 cores, 2.4GHz) x 2
    Memory: 6GB
    HDD: HP ???? (SAS, 72GB, 15krpm) x 2
  Software:
    OS: Red Hat Enterprise Linux 5 (i386)
    Kernel: Linux 2.6.18-53.el5PAE
    Filesystem: ext3, RAID1 (2 disks) by HP Smart Array P400
    Role: Domain controller
    Packages:
      samba-3.0.24-1.54_OSSTECH.el5.i386

NAME             LABEL                    SIZE      FILES      WORDS CACHE%
----             -----                    ----      -----      ----- ------
chimera          chimera           37205308739    2299087   66239072      0

$ /opt/osstech/bin/chimera-crawler --check-updated
...
check updated: statistics: URI: smb://172.16.17.1/xxx_secure
check updated: statistics: directory: 419727
check updated: statistics: file: 4356614
check updated: statistics: target file: 2292586
check updated: statistics: new target file: 2292586
check updated: statistics: updated target file: 0
check updated: statistics: old target file: 0
check updated: statistics: indexed target file (lamed): 2292586 (216672)
check updated: statistics: indexed target file total size (per file): 631326428288 (275377)
check updated: statistics: time (per file): 2111139.210323 (0.920854969158409)
...
check updated: statistics: URI: smb://172.16.17.1/xxx_open
check updated: statistics: directory: 2574
check updated: statistics: file: 12512
check updated: statistics: target file: 6501
check updated: statistics: new target file: 6501
check updated: statistics: updated target file: 0
check updated: statistics: old target file: 0
check updated: statistics: indexed target file (lamed): 6501 (653)
check updated: statistics: indexed target file total size (per file): 4611665112 (709377)
check updated: statistics: time (per file): 22597.299916 (3.47597291432087)
check updated: end: smb://172.16.17.1/xxx_open

real    35562m20.835s
user    2418m17.547s
sys     6721m8.496s

= 24 days 17 hours

2009-04-10
----------------------------------------------------------------------

Chimera Search:
  Hardware: ?
    CPU: ?
    Memory ?
    HDD: ?
  Software:
    OS: CentOS 5 x86-64
    Kernel: Linux 2.6.?? FIXME
    Filesystem: ext3
    Package:
      chimera-0.10.2
      hyperestraier-1.4.13
      qdbm-1.8.77
      libsmbclient 3.2.4

SMB server:
  Hardware: See abobe
  Software: See abobe
    Role: Standalone server

$ nice ionice -c 3 /opt/osstech/bin/chimera-crawler --check-updated
...
check updated: statistics: URI: smb://oooo/dfsroot01-c/syokuin
check updated: statistics: directory: 63241
check updated: statistics: file: 518537
check updated: statistics: target file: 397529
check updated: statistics: new target file: 397422
check updated: statistics: updated target file: 0
check updated: statistics: old target file: 107
check updated: statistics: indexed target file (lamed): 397421 (114)
check updated: statistics: indexed target file total size (per file): 57534566784 (144769)
check updated: statistics: time (per file): 225860.005112 (0.568314218705101)
check updated: end: smb://oooo/dfsroot01-c/syokuin
...
check updated: statistics: URI: smb://oooo/dfsroot02-c/gakko
check updated: statistics: directory: 90494
check updated: statistics: file: 885289
check updated: statistics: target file: 442656
check updated: statistics: new target file: 442655
check updated: statistics: updated target file: 0
check updated: statistics: old target file: 1
check updated: statistics: indexed target file (lamed): 442654 (444)
check updated: statistics: indexed target file total size (per file): 102174516978 (230822)
check updated: statistics: time (per file): 264955.816865 (0.598561894538398)
check updated: end: smb://oooo/dfsroot02-c/gakko

137 時間 30 分 (5 日と 17 時間半)

# /opt/osstech/bin/chimera-crawler --list-index
NAME             LABEL                    SIZE      FILES      WORDS CACHE%
----             -----                    ----      -----      ----- ------
chimera          chimera            5983948463     840177    3608307      0

2008-10-27
----------------------------------------------------------------------

Chimera Search:
  Hardware: LinkStation LS-C500L
    CPU: ARM926EJ-S rev 0 (266 BogoMips)
    Memory 128MB
    HDD: Hitachi HTS72202 (500GB, SATA)
  Software:
    OS: LinkStation Linux 1.02 (0.45)
    Kernel: Linux 2.6.22.7
    Filesystem: XFS
    Package:
      chimera-0.10.1
      hyperestraier-1.4.13
      qdbm-1.8.77

SMB server:
  Hardware: See abobe
  Software: See abobe
    Role: Standalone server

$ time /opt/osstech/bin/chimera-crawler --check-updated
...
check updated: statistics: URI: smb://LS-CL49D/info
check updated: statistics: directory: 22
check updated: statistics: file: 622
check updated: statistics: target file: 141
check updated: statistics: new target file: 141
check updated: statistics: updated target file: 0
check updated: statistics: old target file: 0
check updated: statistics: indexed target file (lamed): 141 (0)
check updated: statistics: indexed target file total size (per file): 7124947 (50531)
check updated: statistics: time (per file): 44.325294 (0.314363787234043)

real    1m15.593s
user    0m50.050s
sys     0m5.210s

$ time /opt/osstech/bin/chimera-crawler --list-index
Using netbios name LS-CL49D.
Using workgroup WORKGROUP.
NAME             LABEL                    SIZE      FILES      WORDS CACHE%
----             -----                    ----      -----      ----- ------
chimera          chimera               9836994        141      11341      0

real    0m30.289s
user    0m26.300s
sys     0m2.270s

2008-10-19
----------------------------------------------------------------------

Chimera Search:
  Hardware: HP ProLiant DL385 G2
    CPU: AMD Opteron 2216 HE (2 cores, 2.4GHz) x 2
    Memory: 6GB
    HDD: HP ???? (SAS, 72GB, 15krpm) x 2
  Software:
    OS: Red Hat Enterprise Linux 5 (i386)
    Kernel: Linux 2.6.18-53.el5PAE
    Filesystem: ext3, RAID1 (2 disks) by HP Smart Array P400
    Packages:
      osstech-chimera-0.10.0-1.el5.noarch
      osstech-hyperestraier-1.4.13-0.12.el5.i386
      osstech-qdbm-1.8.77-0.5.el5.i386

SMB server:
  Hardware: See above (but other host)
  Software: See above
    Role: Domain controller
    Packages:
      samba-3.0.24-1.39_OSSTECH.el5.i386

$ /opt/osstech/bin/chimera-crawler --check-updated
...
chimera:: check updated: statistics: URI: smb://172.16.17.1/xxx_secure
chimera:: check updated: statistics: directory: 53536
chimera:: check updated: statistics: file: 494014
chimera:: check updated: statistics: target file: 254871
chimera:: check updated: statistics: new target file: 175297
chimera:: check updated: statistics: updated target file: 1
chimera:: check updated: statistics: old target file: 79573
chimera:: check updated: statistics: indexed target file (lamed): 175298 (5756)
chimera:: check updated: statistics: indexed target file total size (per file): 39616628745 (225995)
chimera:: check updated: statistics: time (per file): 142853.707206 (0.814919207326952)
chimera:: check updated: end: smb://172.16.17.1/xxx_secure
...
chimera:: check updated: statistics: URI: smb://172.16.17.1/xxx_open
chimera:: check updated: statistics: directory: 225
chimera:: check updated: statistics: file: 190
chimera:: check updated: statistics: target file: 79
chimera:: check updated: statistics: new target file: 79
chimera:: check updated: statistics: updated target file: 0
chimera:: check updated: statistics: old target file: 0
chimera:: check updated: statistics: indexed target file (lamed): 79 (3)
chimera:: check updated: statistics: indexed target file total size (per file): 210609908 (2665948)
chimera:: check updated: statistics: time (per file): 229.600682 (2.90633774683544)
chimera:: check updated: end: smb://172.16.17.1/xxx_open

$ /opt/osstech/bin/chimera-crawler --list-index
NAME             LABEL                    SIZE      FILES      WORDS CACHE%
----             -----                    ----      -----      ----- ------
chimera          chimera            3796558388     254949   20225116      0

2008-07-12
----------------------------------------------------------------------

Chimera Search:
  Hardware:
    CPU: Intel Xeon (3.2GHz) x 2
    Memory: 2GB
    HDD: HP ???? (Ultra320, 146GB, 15krpm)
  Software:
    OS: MIRACLE LINUX 4 (i386)
    Kernel: 2.6.9-34.21AXsm
    Filesystem: xfs, RAID5 (14 disks?) by HP Smart Array 64XX
    Packages:
      osstech-chimera-0.9.5-1.el4
      osstech-hyperestraier-1.4.13-0.3.el4.i386
      osstech-qdbm-1.8.77-0.1.el4.i386

SMB server:
  Hardware: See above
  Software: See above
    Role: Domain controller
    SMB server: Samba 3.0.24-1.48.osstech

$ time /opt/osstech/bin/chimera-crawler --verbose-level 1 --check-updated
...
check updated: statistics: URI: smb://yokofs2/IVCS
check updated: statistics: directory: 7238
check updated: statistics: file: 75515
check updated: statistics: target file: 26146
check updated: statistics: new target file: 26146
check updated: statistics: updated target file: 0
check updated: statistics: old target file: 0
check updated: statistics: indexed target file (lamed): 26146 (552)
check updated: statistics: indexed target file total size (per file): 9083844891 (347427)
check updated: statistics: time (per file): 3346.248432 (0.127983187944619)
check updated: end: smb://yokofs2/IVCS

real    55m48.393s
user    34m25.631s
sys     10m24.287s

$ /opt/osstech/bin/chimera-crawler --list-index
NAME                 LABEL                        SIZE      FILES      WORDS
----                 -----                        ----      -----      -----
chimera              chimera               147653344.0      26146     326931

$ /opt/osstech/bin/chimera-crawler --verbose-level 1 --check-updated
...
check updated: statistics: URI: smb://yokofs2/eigyo
check updated: statistics: directory: 15015
check updated: statistics: file: 135490
check updated: statistics: target file: 61579
check updated: statistics: new target file: 61579
check updated: statistics: updated target file: 0
check updated: statistics: old target file: 0
check updated: statistics: indexed target file (lamed): 61579 (480)
check updated: statistics: indexed target file total size (per file): 13592798411 (220737)
check updated: statistics: time (per file): 8448.072963 (0.137190811201871)
check updated: end: smb://yokofs2/eigyo
(2 hours 20 miniutes)
...
check updated: statistics: URI: smb://yokofs2/genka
check updated: statistics: directory: 714
check updated: statistics: file: 5735
check updated: statistics: target file: 5212
check updated: statistics: new target file: 5212
check updated: statistics: updated target file: 0
check updated: statistics: old target file: 0
check updated: statistics: indexed target file (lamed): 5212 (73)
check updated: statistics: indexed target file total size (per file): 779818030 (149619)
check updated: statistics: time (per file): 1185.418906 (0.227440311972371)
check updated: end: smb://yokofs2/genka
(fixme hours fixme minites)

real    160m35.897s
user    83m41.521s
sys     33m14.176s

$ /opt/osstech/bin/chimera-crawler --list-index
NAME                 LABEL                        SIZE      FILES      WORDS
----                 -----                        ----      -----      -----
chimera              chimera               611097315.0      92937    1060954

2008-07-11
----------------------------------------------------------------------

Chimera Search:
  Hardware:
    CPU: Intel Celeron (prescott?, 2.66GHz)
    Memory: 1.5GB
    HDD: Western Digital WD800JD-19MS (SATA, 80GB, 7200rpm)
  Software:
    OS: CentOS 4 (i386)
    Kernel: Linux 2.6.9-55.0.2.EL
    Filesystem: ext3
    Packages:
      osstech-chimera-0.9.5-1.el4
      osstech-hyperestraier-1.4.13-0.3.el4.i386
      osstech-qdbm-1.8.77-0.1.el4.i386

SMB server:
  Hardware:
    CPU: AMD Athlon 64 X2 3800+ (2 cores, 2GHz)
    Memory: 2GB (DDR2, 266MHz)
    HDD: Western Digital WD5000AAKS-0 (SATA, 500GB, 7200rpm)
  Software:
    Role: Domain controller
    OS: Debian GNU Linux 4.0 (etch, x86-64)
    Kernel: Linux 2.6.22-2-amd64
    Filesystem: ext3, LVM2, RAID1 (1 disk) by Linux md
    SMB server: Samba 3.0.30-1.1.osstech

Misc:
  100Base-TX <-> 100Base-TX

$ /opt/osstech/bin/chimera-crawler --verbose-level 1 --check-updated
...
check updated: statistics: URI: smb://ocean/project
check updated: statistics: directory: 19921
check updated: statistics: file: 86599
check updated: statistics: target file: 7795
check updated: statistics: new target file: 7406
check updated: statistics: updated target file: 0
check updated: statistics: old target file: 389
check updated: statistics: indexed target file (lamed): 7406 (193)
check updated: statistics: indexed target file total size (per file): 1222967854 (165132)
check updated: statistics: time (per file): 864.241261 (0.116694742236025)
check updated: end: smb://ocean/project

$ /opt/osstech/bin/chimera-crawler --list-index
NAME                 LABEL                        SIZE      FILES      WORDS
----                 -----                        ----      -----      -----
chimera              chimera                87962452.0       7795     434259

2008-03-20
----------------------------------------------------------------------

Chimera Search:
  CPU: Intel Celeron (prescott?, 2.66GHz)
  Memory: 1.5GB
  HDD: Western Digital WD800JD-19MS (SATA, 80GB, 7200rpm)
  OS: CentOS 4 (i386)
  Kernel: Linux 2.6.9-??.EL
  Filesystem: ext3
  Packages:
    osstech-chimera-0.9.3-0.2.el4
    osstech-hyperestraier-1.4.13-0.2.el4.i386
    osstech-qdbm-1.8.77-0.1.el4.i386

SMB server:
  CPU: AMD Athlon 64 X2 3800+ (2 cores, 2GHz)
  Memory: 2GB (DDR2, 266MHz)
  HDD: Hitachi ???? (SATA, 250GB, 7200rpm)
  OS: Debian GNU Linux 4.0 (etch, x86-64)
  Kernel: Linux 2.6.22-2-amd64
  Filesystem: ext3, LVM2, RAID5 (3 disks) by Adaptec 2420SA (PCI 32bit)
  SMB server: Samba 3.0.24-1.3?.osstech

Misc:
  100Base-TX

$ time /opt/osstech/bin/chimera-crawler --verbose-level 1 --check-updated
...
check updated: statistics: URI: smb://ocean/project
check updated: statistics: directory: 17142
check updated: statistics: file: 46756
check updated: statistics: target file: 6647
check updated: statistics: new target file: 6647
check updated: statistics: updated target file: 0
check updated: statistics: old target file: 0
check updated: statistics: indexed target file (lamed): 6647 (32)
check updated: statistics: indexed target file total size (per file): 890801076 (134015)
check updated: statistics: time (per file): 682.60243 (0.102693309763803)
check updated: end: smb://ocean/project

real    11m22.969s
user    6m10.839s
sys     1m10.063s

$ /opt/osstech/bin/chimera-crawler --list-index
NAME                 LABEL                        SIZE      FILES      WORDS
----                 -----                        ----      -----      -----
chimera              chimera                54742229.0       6647     367464

