Setup

Pre-requisites

  • Hitachi Content Platform (HCP)
  • A Tenant-level user account having read/write/delete access to an HCP Namespace

Dependencies

You need to have at least Python 3.4.3 installed to run hcpt.

It depends on the hcpsdk to access HCP.

You might want to use a virtual environment to fence the dependency from your primary Python environment…

Installation

Make sure you have Python 3.4.3 (or better) installed

In case it’s not installed, get it here.

There are two ways to install hcpt:

  1. system-wide

    • Install hcpt by running:

      $ pip install hcpt

    -or-

    • Get the source from gitlab.com either

      • by downloading the source archive, or

      • by cloning the repository:

        $ git clone https://gitlab.com/simont3/hcptool

    • Install locally, including the dependency:

      $ python setup.py install

  2. in a virtual environment

WINDOWS

  • Create a fresh virtual environment:

    C:\>\Python35\Tools\scripts\pyvenv.py c:\temp\_venv_hcpt

  • Activate the virtual environment:

    C:\>cd temp

C:\temp>\temp\_venv_hcpt\Scripts\activate.bat
(_venv_hcpt) C:\temp>

  • Install hcpt:

    (_venv_hcpt) C:\temp>pip install hcpt
Collecting hcpt
  Downloading hcpt-1.0.19.tar.gz
Collecting hcpsdk>=0.9.3.post0 (from hcpt)
  Downloading hcpsdk-0.9.3.post4.tar.gz
Collecting dnspython3==1.12.0 (from hcpsdk>=0.9.3.post0->hcpt)
  Downloading dnspython3-1.12.0.zip (226kB)
    100% |################################| 229kB 1.3MB/s
Collecting alabaster>=0.7.1 (from hcpsdk>=0.9.3.post0->hcpt)
  Downloading alabaster-0.7.6.tar.gz
Installing collected packages: dnspython3, alabaster, hcpsdk, hcpt
  Running setup.py install for dnspython3
  Running setup.py install for alabaster
  Running setup.py install for hcpsdk
  Running setup.py install for hcpt
Successfully installed alabaster-0.7.6 dnspython3-1.12.0 hcpt-1.0.19 hcpsdk-0.9.3.post4

Now you can run hcpt as long as you have the virtual environment activated:

C:\temp>hcpt
blablabla

Linux

  • Create a fresh virtual environment:

    $ pyvenv _venv_hcpt

  • Activate the virtual environment:

    $ source _venv_hcpt/bin/activate

  • Install hcpt:

    (_venv_hcpt) $ pip install hcpt
Collecting hcpt
  Downloading hcpt-1.0.19.tar.gz
Collecting hcpsdk>=0.9.3.post0 (from hcpt)
  Downloading hcpsdk-0.9.3.post4.tar.gz
Collecting dnspython3==1.12.0 (from hcpsdk>=0.9.3.post0->hcpt)
  Downloading dnspython3-1.12.0.zip (226kB)
    100% |################################| 229kB 7.4MB/s
Collecting alabaster>=0.7.1 (from hcpsdk>=0.9.3.post0->hcpt)
  Downloading alabaster-0.7.6.tar.gz
Installing collected packages: dnspython3, alabaster, hcpsdk, hcpt
  Running setup.py install for dnspython3
  Running setup.py install for alabaster
  Running setup.py install for hcpsdk
  Running setup.py install for hcpt
Successfully installed alabaster-0.7.6 dnspython3-1.12.0 hcpt-1.0.19 hcpsdk-0.9.3.post4

Now you can run hcpt as long as you have the virtual environment activated:

(_venv_hcpt) $ hcpt
blablabla

Verification

  • Make sure that you have access to an HCP system and valid user credentials for an authenticated namespace at hand. The test may also be run against the default namespace, where’s no need to have user credentials (simply provide random user / password).

  • Get a command prompt

  • If ever possible, switch of the local machine’s DNS cache:

    c:> net stop dnscache

  • Run hcpt in test mode:

    c:> hcpt -i5 --user <user> --password <password> test --cluster \
          ns.tenant.hcp.vm.loc --dir /rest/hcpt_test --file <filename>
          --structure 10 100

    (if the target namespace has ‘versioning’ eabled, add –versionedNS to the command above!)

hcpt will…

  1. calculate access tokens for both namespace and
  2. MAPI access
  3. feed file <filename> 100 times in each of 10 subdirectories of /rest/hcpt_test within namespace ‘ns’ in tenant ‘tenant’ within an HCP with a DNS name of ‘hcp.vm.loc’
  4. discover a list of all objects and directories, beginning at /rest/hcpt_test. This list will be provided as 2 files: hcpt_test.csv (to be loaded into MS Excel) and hcpt_list.<timestamp>.db, a Sqlite3-database that will be used as base for the nexts steps.
  5. update the database file with a new retention setting to be processed in the next step.
  6. update HCP with the new retention settings for the objects selected in the database in the prior step.
  7. delete (purge) all the objects and the directories where they have been stored.

At the end, there will be a logfile (hcp_test.log) showing all the output that has been showed on the screen, an Excel-importable file (hcp_test.csv) containing a list of all objects and directories found in 4. and two Sqlite3 database files, containing the results of 4. and 7.

test run output:

02/12 09:25:49 [INFO    ] hcpt test v.0.9.3 (2011-07-07/Sm)
02/12 09:25:49 [INFO    ]     Logfile: hcpt_test.log
02/12 09:25:49 [INFO    ]      Target: https://test.m.hcp73.archivas.com/rest/hcpt_test15
02/12 09:25:49 [INFO    ] DataAccAcnt: n
02/12 09:25:49 [INFO    ]  Ingestfile: readme.txt
02/12 09:25:49 [INFO    ]  Started at: Fri, 12.02., 09:25:49
02/12 09:25:49 [INFO    ] =====================================================
02/12 09:25:49 [INFO    ]
02/12 09:25:49 [INFO    ] -------------------------------------------------
02/12 09:25:49 [INFO    ] Test 1: generate a cookie for a DataAccessAccount
02/12 09:25:49 [INFO    ] -------------------------------------------------
02/12 09:25:49 [INFO    ] hcp-ns-auth=bg==:1dc7fed37e11b35093d311ef66928ad9
02/12 09:25:49 [INFO    ]
02/12 09:25:49 [INFO    ] -----------------------------------------
02/12 09:25:49 [INFO    ] Test 2: generate a cookie for MAPI access
02/12 09:25:49 [INFO    ] -----------------------------------------
02/12 09:25:49 [INFO    ] hcp-api-auth=bg==:1dc7fed37e11b35093d311ef66928ad9
02/12 09:25:49 [INFO    ]
02/12 09:25:49 [INFO    ] ------------------------------------
02/12 09:25:49 [INFO    ] Test 3: ingest some objects into HCP
02/12 09:25:49 [INFO    ] ------------------------------------
02/12 09:25:49 [INFO    ] hcpt load v.1.1.0 (2015-02-20/Sm)
02/12 09:25:49 [INFO    ]     Logfile: hcpt_test.log
02/12 09:25:49 [INFO    ] Loginterval: 5 sec.
02/12 09:25:49 [INFO    ]      Target: https://test.m.hcp73.archivas.com/rest/hcpt_test15/
02/12 09:25:49 [INFO    ] DataAccAcnt: n
02/12 09:25:49 [INFO    ]  Ingestfile: readme.txt
02/12 09:25:49 [INFO    ]    Filesize: 1012 Bytes (= 1012.00 Byte)
02/12 09:25:49 [INFO    ]   Retention: 0
02/12 09:25:49 [INFO    ]   Structure: [10, 100] = 1,000 PUTs (= 988.28 Kbyte)
02/12 09:25:49 [INFO    ]     Threads: 30
02/12 09:25:49 [INFO    ]  Started at: Fri, 12.02., 09:25:49
02/12 09:25:49 [INFO    ] --------------------------------------------------
02/12 09:25:49 [INFO    ] MonitorThread started monitoring Worker.Qbox.qsize()
02/12 09:25:54 [INFO    ] files sent: 976, errors: 0, PUTs/sec.: 194.5
02/12 09:25:55 [INFO    ] MonitorThread exits on Worker.Qbox.qsize() == 0
02/12 09:25:55 [INFO    ] --------------------------------------------------
02/12 09:25:55 [INFO    ]  Started at: Fri, 12.02., 09:25:49
02/12 09:25:55 [INFO    ]    Ended at: Fri, 12.02., 09:25:55
02/12 09:25:55 [INFO    ]     Runtime: 5.12 Sekunden
02/12 09:25:55 [INFO    ]     Success: 1000
02/12 09:25:55 [INFO    ]      Errors: 0
02/12 09:25:55 [INFO    ]  Statistics:
02/12 09:25:55 [INFO    ]   Obj./Sec.: 195.2
02/12 09:25:55 [INFO    ]    Transfer: 192.88 Kbyte/sec.
02/12 09:25:55 [INFO    ]       t-sum:   147,216 ms
02/12 09:25:55 [INFO    ]   t-average:       147 ms
02/12 09:25:55 [INFO    ]       t-min:        21 ms
02/12 09:25:55 [INFO    ]       t-max:       359 ms
02/12 09:25:55 [INFO    ]    t-median:       129 ms
02/12 09:25:55 [INFO    ]  t-90%-line:       234 ms
02/12 09:25:55 [INFO    ]
02/12 09:25:55 [INFO    ] ---------------------------------------
02/12 09:25:55 [INFO    ] Test 4: list the newly ingested objects
02/12 09:25:55 [INFO    ] ---------------------------------------
02/12 09:25:55 [STARTUP ] hcpt list v.1.4.6 (2015-05-20/Sm)
02/12 09:25:55 [STARTUP ]     Logfile: hcpt_test.log
02/12 09:25:55 [STARTUP ] Loginterval: 5 sec.
02/12 09:25:55 [STARTUP ]    Database: hcpt_list.20161202092549.db
02/12 09:25:55 [STARTUP ]              (will include deleted objects)
02/12 09:25:55 [STARTUP ]      Target: https://test.m.hcp73.archivas.com/rest/hcpt_test15
02/12 09:25:55 [STARTUP ] DataAccAcnt: n
02/12 09:25:55 [STARTUP ]     Threads: 30
02/12 09:25:55 [STARTUP ]     FindQue: -1
02/12 09:25:55 [STARTUP ] dbWriterQue: -1
02/12 09:25:55 [STARTUP ]   Verbosity: 1
02/12 09:25:55 [STARTUP ]  Started at: Fri, 12.02., 09:25:55
02/12 09:25:55 [STARTUP ] -------------------------------------------------------------------------------
02/12 09:25:55 [INFO    ] *** monitor started ***
02/12 09:25:55 [INFO    ] *** dbWriter started ***
02/12 09:25:56 [INFO    ] *** Discovery finished after 00:00:01.2 ***
02/12 09:25:56 [INFO    ] *** dbWriter finished after 00:00:01.3***
02/12 09:26:00 [INFO    ] Dirs:11; Objs:1,000; Err/Warn:0/0; dbWrites:1,011 [QF:0; Qdb:0]
02/12 09:26:00 [INFO    ] *** monitor exits ***
02/12 09:26:00 [STOPDOWN] -------------------------------------------------------------------------------
02/12 09:26:00 [STOPDOWN]  Started at: Fri, 12.02., 09:25:55
02/12 09:26:00 [STOPDOWN]    Ended at: Fri, 12.02., 09:26:00
02/12 09:26:00 [STOPDOWN]     Runtime: 00:00:05.1 h:m:s.ms
02/12 09:26:00 [STOPDOWN]       Found: 11 Directories, 1,000 Objects
02/12 09:26:00 [STOPDOWN]    Warnings: 0
02/12 09:26:00 [STOPDOWN]      Errors: 0
02/12 09:26:00 [INFO    ]
02/12 09:26:00 [INFO    ] -------------------------------------------------------
02/12 09:26:00 [INFO    ] Test 5: prepare the database file for retention changes
02/12 09:26:00 [INFO    ] -------------------------------------------------------
02/12 09:26:00 [INFO    ] *** database update begins ***
02/12 09:26:00 [INFO    ] *** changing all object's retention to 'N+1s' ***
02/12 09:26:00 [INFO    ] *** database update finished ***
02/12 09:26:00 [INFO    ]
02/12 09:26:00 [INFO    ] ----------------------------------------------------------
02/12 09:26:00 [INFO    ] Test 6: change the retention of the newly ingested objects
02/12 09:26:00 [INFO    ] ----------------------------------------------------------
02/12 09:26:00 [STARTUP ] hcpt retention v.0.9.5 (2011-07-02/Sm)
02/12 09:26:00 [STARTUP ]     Logfile: hcpt_test.log
02/12 09:26:00 [STARTUP ] Loginterval: 5 sec.
02/12 09:26:00 [STARTUP ]    Database: hcpt_list.20161202092549.db (1.4.6/2015-05-20/Sm)
02/12 09:26:00 [STARTUP ]      Target: https://test.m.hcp73.archivas.com
02/12 09:26:00 [STARTUP ] DataAccAcnt: n
02/12 09:26:00 [STARTUP ]     Threads: 30
02/12 09:26:00 [STARTUP ]   Verbosity: 1
02/12 09:26:00 [STARTUP ]  Started at: Fri, 12.02., 09:26:00
02/12 09:26:00 [STARTUP ] -------------------------------------------------------------------------------
02/12 09:26:00 [INFO    ] *** monitor started ***
02/12 09:26:00 [INFO    ] *** Changing retentions started ***
02/12 09:26:05 [INFO    ] Changed: 834, Errors=0, Warnings=0
02/12 09:26:07 [INFO    ] *** Changing retentions finished after 00:00:07.45 ***
02/12 09:26:10 [INFO    ] Changed: 1,000, Errors=0, Warnings=0
02/12 09:26:10 [INFO    ] *** monitor exits ***
02/12 09:26:10 [STOPDOWN] -------------------------------------------------------------------------------
02/12 09:26:10 [STOPDOWN]  Started at: Fri, 12.02., 09:26:00
02/12 09:26:10 [STOPDOWN]    Ended at: Fri, 12.02., 09:26:10
02/12 09:26:10 [STOPDOWN]     Runtime: 00:00:10.1 h:m:s.ms
02/12 09:26:10 [STOPDOWN]     Changed: 1000
02/12 09:26:10 [STOPDOWN]    Warnings: 0
02/12 09:26:10 [STOPDOWN]      Errors: 0
02/12 09:26:10 [INFO    ]
02/12 09:26:10 [INFO    ] -----------------------------------------
02/12 09:26:10 [INFO    ] Test 7: delete the newly ingested objects
02/12 09:26:10 [INFO    ] -----------------------------------------
02/12 09:26:10 [STARTUP ] hcpt unload v.1.1.12 (2015-02-27/Sm)
02/12 09:26:10 [STARTUP ]     Logfile: hcpt_test.log
02/12 09:26:10 [STARTUP ] Loginterval: 5 sec.
02/12 09:26:10 [STARTUP ]    Database: hcpt_unload.20161202092549.db
02/12 09:26:10 [STARTUP ]              (will be kept when finished)
02/12 09:26:10 [STARTUP ]      Target: https://test.m.hcp73.archivas.com/rest/hcpt_test15
02/12 09:26:10 [STARTUP ] DataAccAcnt: n
02/12 09:26:10 [STARTUP ]     Threads: 30
02/12 09:26:10 [STARTUP ]   QueueSize: -1
02/12 09:26:10 [STARTUP ]       Purge: yes
02/12 09:26:10 [STARTUP ]  Privileged: yes (Reason: 'hcpthcptcmds')
02/12 09:26:10 [STARTUP ]   Verbosity: 1
02/12 09:26:10 [STARTUP ]  DeleteMode: REQUESTED - Objects and Directories
02/12 09:26:10 [STARTUP ]  Started at: Fri, 12.02., 09:26:10
02/12 09:26:10 [STARTUP ] -------------------------------------------------------------------------------
02/12 09:26:10 [WARNING ] MonitorThread started - monitoring Worker Threads
02/12 09:26:10 [INFO    ] *** dbWriter started ***
02/12 09:26:10 [WARNING ] ***Discovery finished***
02/12 09:26:15 [INFO    ] Dirs: 11/0, Objects: 1000/605 (found/deleted), Errors: 0 [Q:0/365/0 (find/del/dbW)]
02/12 09:26:17 [WARNING ] ***Object deletion finished***
02/12 09:26:18 [INFO    ] Thread-1133: http (DELETE) error: 403:/rest/hcpt_test15 (re-scheduled for later deletion)
02/12 09:26:18 [WARNING ] ***Directory deletion finished***
02/12 09:26:20 [INFO    ] Dirs: 11/11, Objects: 1000/1000 (found/deleted), Errors: 0 [Q:0/0/0 (find/del/dbW)]
02/12 09:26:20 [WARNING ] MonitorThread exits - all Worker Threads ended
02/12 09:26:20 [STOPDOWN] -------------------------------------------------------------------------------
02/12 09:26:20 [STOPDOWN]  Started at: Fri, 12.02., 09:26:10
02/12 09:26:20 [STOPDOWN]    Ended at: Fri, 12.02., 09:26:20
02/12 09:26:20 [STOPDOWN]     Runtime: 10.01 Sekunden
02/12 09:26:20 [STOPDOWN]       Found: 11 Directories, 1000 Objects
02/12 09:26:20 [STOPDOWN]     Deleted: 11 Directories, 1000 Objects
02/12 09:26:20 [STOPDOWN]      Errors: 0
02/12 09:26:20 [INFO    ]
02/12 09:26:20 [INFO    ] =====================================================
02/12 09:26:20 [INFO    ]  Started at: Fri, 12.02., 09:25:49
02/12 09:26:20 [INFO    ]    Ended at: Fri, 12.02., 09:26:20
02/12 09:26:20 [INFO    ]     Runtime: 30.19 Sekunden