PMD deployment issues and possible errata

Dear Sir or Madam,

I’m currently working on the GlasDigital project.
Since we have an private IP address for database within the University of Jena, which is only accessible via VPN, it is not possible to issue a Let’s Encrypt certificates.
I used a self-signed certificate instead for now.

Our domain name is:
glasdigi.cms.uni-jena.de

Before I put it online, I’m currently using a test domain name I purchased from Namecheap and testing it under Virtual Machine.
My test domain name is „nana-steelfan.xyz“.
I use Cloudflare to manage the DNS setting.
I set another two A records:


sso.nana-steelfan.xyz is used for KEYCLOAK_URL and ontodocker.nana-steelfan.xyz is used for APPLICATION_URL

I wonder if this is the right way to configure the domain name of SSO and database interface?
So I could ask the university to handle that.

I follow the tutorial under III. Reverse proxy with independently retrieved certificates section & 4. Make the service available by adding it to the reverse proxy section

I replace the following codes in keycloak.conf:

ssl_certificate /etc/letsencrypt/live/[URL]/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/[URL]/privkey.pem;
include /etc/letsencrypt/options-ssl-nginx.conf;
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

with

ssl_certificate /run/secrets/cert.pem;
ssl_certificate_key /run/secrets/key.pem;
include ./conf.d/ssl.incl;
ssl_dhparam /run/secrets/dhparam.pem;

follow the rest of steps, I was able to access Keycloak under the [KEYCLOAK_URL].

(Thank you for updating the Ontodocker tutorial)
I saved the Initial Access Token as ia.jwt and replaced KEYCLOAK_URL, REALM_NAME in provider_info.json

My KEYCLOAK_URL is „sso.nana-steelfan.xyz“ and REALM_NAME is „master“, and APPLICATION_URL is „ontodocker.nana-steelfan.xyz

I can access [KEYCLOAK_URL] but when I navigate to the URL of [ONTODOCKER_URL]/[APPLICATION_URL], it shows

Internal Server Error

I think I did something wrong, but I don’t know where😢
Here are some screenshots of Keycloak setting and Ontodocker status:



It would be very helpful if there were detailed tutorials about how to connect OntoDocker to the IDP.

Possible errata:

in PMD Core Components - PMD Deployment guide

cd pmd server

should be

cd pmd-server

in Reverse Proxy - PMD Deployment guide

sed „s/[URL]/foo.bar.org/“ compose-templates/docker-compose-nginx-ssl.yml > docker-compose.yml

should be

sed „s/[URL]/foo.bar.org/“ data/nginx/nginx_ssl.conf.template > data/nginx/nginx_ssl.conf

and

cp data/nginx/nginx_ssl.conf.template data/nginx/nginx_ssl.conf

should be replace by

cp compose-templates/docker-compose-nginx-ssl.yml docker-compose.yml

1 Like

Dear Ya-Fan,

thank you for your post, I’m not sure if I can give a helpful reply to all issues right away, But a first few comments/suggestions:

  • At the moment I would not recommend running your own Keycloak instance but rather ask for an initial access token from the test instance (sso-dev.material-digital.de)

  • It seems like there is an issue with ontodoocker, so you should get more information by looking at the logs of this app (docker-compose logs ontodocker)

    • even though the logs are missing I would assume that ontodocker might complain that the connection to the sso is not secure due to the self-signed certificate
  • to my knowledge, there are some pending changes to ontodocker and it might be worthwhile waiting for those before you setup your instance

  • if you find issues/errata in the deployment guide, feel free to open the issue in the repository https://github.com/materialdigital/pmd-server/tree/main/docs or directly create a merge request to speed up the process

Best Joerg

1 Like

Questions regarding OntoDocker can be answered by @henkbirkholz and @jannis.grundmann

Dear Schaarschmidt,
Thank you for your quick reply.
Indeed, there are some issues with ontodocker.

Sorry to bother, but how to ask for an initial access token from sso-dev.material-digital.de ?
I cannot log in with the pmd account.

And here’s the complete log message:

yafan@test:~/Desktop/pmd-server/ontodocker$ docker-compose up
Creating network „ontodocker_default“ with the default driver
Creating ontodocker_blazegraph_1 … done
Creating ontodocker_jena_1 … done
Creating ontodocker_ontodocker_1 … done
Attaching to ontodocker_blazegraph_1, ontodocker_jena_1, ontodocker_ontodocker_1
blazegraph_1 |
blazegraph_1 | Blazegraph init process done. Ready for start up.
blazegraph_1 |
blazegraph_1 | java -Xms512m -Xmx1g -jar /usr/bin/blazegraph.jar
blazegraph_1 | INFO: com.bigdata.util.config.LogUtil: Configure: jar:file:/usr/bin/blazegraph.jar!/log4j.properties
blazegraph_1 |
blazegraph_1 | BlazeGraph™ Graph Engine
blazegraph_1 |
blazegraph_1 | Flexible
blazegraph_1 | Reliable
blazegraph_1 | Affordable
blazegraph_1 | Web-Scale Computing for the Enterprise
blazegraph_1 |
blazegraph_1 | Copyright SYSTAP, LLC DBA Blazegraph 2006-2016. All rights reserved.
blazegraph_1 |
blazegraph_1 | 5b865249211e
blazegraph_1 | Tue Oct 05 00:01:36 GMT 2021
blazegraph_1 | Linux/5.8.0-59-generic amd64
blazegraph_1 | Intel(R) Core™ i7-10850H CPU @ 2.70GHz Family 6 Model 165 Stepping 2, GenuineIntel #CPU=2
blazegraph_1 | Oracle Corporation 1.8.0_191
blazegraph_1 | freeMemory=506766232
blazegraph_1 | buildVersion=2.1.5
blazegraph_1 | gitCommit=cb08991909034b5fba53c16f5c495e72ab011933
blazegraph_1 |
blazegraph_1 | Dependency License
blazegraph_1 | ICU http://source.icu-project.org/repos/icu/icu/trunk/license.html
blazegraph_1 | bigdata-ganglia Apache License, Version 2.0
blazegraph_1 | blueprints-core https://github.com/tinkerpop/blueprints/blob/master/LICENSE.txt
blazegraph_1 | colt http://acs.lbl.gov/software/colt/license.html
blazegraph_1 | commons-codec Apache License, Version 2.0
blazegraph_1 | commons-fileupload Apache License, Version 2.0
blazegraph_1 | commons-io Apache License, Version 2.0
blazegraph_1 | commons-logging Apache License, Version 2.0
blazegraph_1 | dsiutils http://www.gnu.org/licenses/lgpl-2.1.html
blazegraph_1 | fastutil Apache License, Version 2.0
blazegraph_1 | flot http://www.opensource.org/licenses/mit-license.php
blazegraph_1 | high-scale-lib http://creativecommons.org/licenses/publicdomain
blazegraph_1 | httpclient Apache License, Version 2.0
blazegraph_1 | httpclient-cache Apache License, Version 2.0
blazegraph_1 | httpcore Apache License, Version 2.0
blazegraph_1 | httpmime Apache License, Version 2.0
blazegraph_1 | jackson-core Apache License, Version 2.0
blazegraph_1 | jetty Apache License, Version 2.0
blazegraph_1 | jquery https://github.com/jquery/jquery/blob/master/MIT-LICENSE.txt
blazegraph_1 | jsonld https://raw.githubusercontent.com/jsonld-java/jsonld-java/master/LICENCE
blazegraph_1 | log4j Apache License, Version 2.0
blazegraph_1 | lucene Apache License, Version 2.0
blazegraph_1 | nanohttp http://elonen.iki.fi/code/nanohttpd/#license
blazegraph_1 | rexster-core https://github.com/tinkerpop/rexster/blob/master/LICENSE.txt
blazegraph_1 | river Apache License, Version 2.0
blazegraph_1 | semargl https://github.com/levkhomich/semargl/blob/master/LICENSE
blazegraph_1 | servlet-api Apache License, Version 2.0
blazegraph_1 | sesame http://www.openrdf.org/download.jsp
blazegraph_1 | slf4j SLF4J License
blazegraph_1 | zookeeper Apache License, Version 2.0
blazegraph_1 |
jena_1 | mv: cannot stat ‚/jena-fuseki/extra/’: No such file or directory
ontodocker_1 | [uWSGI] getting INI configuration from uwsgi.ini
ontodocker_1 | *** Starting uWSGI 2.0.19.1 (64bit) on [Tue Oct 5 00:01:36 2021] ***
ontodocker_1 | compiled with version: 8.3.0 on 04 October 2021 11:16:34
ontodocker_1 | os: Linux-5.8.0-59-generic #66~20.04.1-Ubuntu SMP Thu Jun 17 11:14:10 UTC 2021
ontodocker_1 | nodename: 6a32932aaebe
ontodocker_1 | machine: x86_64
ontodocker_1 | clock source: unix
ontodocker_1 | detected number of CPU cores: 2
ontodocker_1 | current working directory: /app
ontodocker_1 | detected binary path: /usr/local/bin/uwsgi
ontodocker_1 | !!! no internal routing support, rebuild with pcre support !!!
ontodocker_1 | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
ontodocker_1 | your memory page size is 4096 bytes
ontodocker_1 | detected max file descriptor number: 1048576
ontodocker_1 | lock engine: pthread robust mutexes
ontodocker_1 | thunder lock: disabled (you can enable it with --thunder-lock)
ontodocker_1 | uwsgi socket 0 bound to TCP address :8000 fd 3
ontodocker_1 | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
ontodocker_1 | Python version: 3.7.3 (default, Jan 22 2021, 20:04:44) [GCC 8.3.0]
ontodocker_1 | *** Python threads support is disabled. You can enable it with --enable-threads ***
ontodocker_1 | Python main interpreter initialized at 0x55b90d8d1f10
ontodocker_1 | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
ontodocker_1 | your server socket listen backlog is limited to 100 connections
ontodocker_1 | your mercy for graceful operations on workers is 60 seconds
ontodocker_1 | mapped 145808 bytes (142 KB) for 1 cores
ontodocker_1 | *** Operational MODE: single process ***
jena_1 | cp: cannot stat '/jena-fuseki/config/data/
‘: No such file or directory
jena_1 | /data/fuseki/config.ttl already exists. Leaving as-is.
ontodocker_1 | *** WARNING: you are running uWSGI as root !!! (use the --uid flag) ***
ontodocker_1 | *** uWSGI is running in multiple interpreter mode ***
ontodocker_1 | spawned uWSGI master process (pid: 1)
ontodocker_1 | spawned uWSGI worker 1 (pid: 7, cores: 1)
ontodocker_1 | unable to stat() RELOAD, events will be triggered as soon as the file is created
ontodocker_1 | mounting run:ontoapp on /
blazegraph_1 | WARN : NanoSparqlServer.java:517: Starting NSS
blazegraph_1 | WARN : ServiceProviderHook.java:171: Running.
ontodocker_1 | Traceback (most recent call last):
ontodocker_1 | File „./run.py“, line 3, in
ontodocker_1 | ontoapp = create_app()
ontodocker_1 | File „./app/init.py“, line 45, in create_app
ontodocker_1 | oidc.init_app(app)
ontodocker_1 | File „/usr/local/lib/python3.7/dist-packages/flask_oidc/init.py“, line 132, in init_app
ontodocker_1 | secrets = self.load_secrets(app)
ontodocker_1 | File „/usr/local/lib/python3.7/dist-packages/flask_oidc/init.py“, line 196, in load_secrets
ontodocker_1 | return _json_loads(open(content, ‚r‘).read())
ontodocker_1 | File „/usr/local/lib/python3.7/dist-packages/flask_oidc/init.py“, line 51, in _json_loads
ontodocker_1 | return json.loads(content)
ontodocker_1 | File „/usr/lib/python3.7/json/init.py“, line 348, in loads
ontodocker_1 | return _default_decoder.decode(s)
ontodocker_1 | File „/usr/lib/python3.7/json/decoder.py“, line 337, in decode
ontodocker_1 | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
ontodocker_1 | File „/usr/lib/python3.7/json/decoder.py“, line 355, in raw_decode
ontodocker_1 | raise JSONDecodeError(„Expecting value“, s, err.value) from None
ontodocker_1 | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
ontodocker_1 | unable to load app 0 (mountpoint=’/’) (callable not found or import error)
**ontodocker_1 | *** no app loaded. going in full dynamic mode *****
blazegraph_1 | serviceURL: http://172.26.0.2:9999
blazegraph_1 |
blazegraph_1 |
blazegraph_1 | Welcome to the Blazegraph™ Database.
blazegraph_1 |
blazegraph_1 | Go to http://172.26.0.2:9999/blazegraph/ to get started.
jena_1 | 00:01:38 INFO Server :: Apache Jena Fuseki 4.0.0
jena_1 | 00:01:38 INFO Config :: FUSEKI_HOME=/jena-fuseki
jena_1 | 00:01:38 INFO Config :: FUSEKI_BASE=/data/fuseki
jena_1 | 00:01:38 INFO Config :: Shiro file: file:///data/fuseki/shiro.ini
jena_1 | 00:01:38 INFO Config :: Configuration file: /data/fuseki/config.ttl
jena_1 | 00:01:39 INFO Config :: Load configuration: file:///data/fuseki/configuration/Tensile_Test.ttl
jena_1 | 00:01:39 INFO Server :: Path = /Tensile_Test
jena_1 | 00:01:39 INFO Server :: Path = /dataset
jena_1 | 00:01:39 INFO Server :: System
jena_1 | 00:01:39 INFO Server :: Memory: 4.0 GiB
jena_1 | 00:01:39 INFO Server :: Java: 11.0.11
jena_1 | 00:01:39 INFO Server :: OS: Linux 5.8.0-59-generic amd64
jena_1 | 00:01:39 INFO Server :: PID: 7
jena_1 | 00:01:39 INFO Server :: Started 2021/10/05 00:01:39 UTC on port 3030
blazegraph_1 | WARN : MapgraphServletProxy.java:67: Running without GPU Acceleration. See https://www.blazegraph.com/product/gpu-accelerated/.
ontodocker_1 | — no python application found, check your startup logs for errors —
ontodocker_1 | [pid: 7|app: -1|req: -1/1] 10.243.191.131 () {44 vars in 703 bytes} [Tue Oct 5 00:06:39 2021] GET / => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (0 switches on core 0)
ontodocker_1 | — no python application found, check your startup logs for errors —
ontodocker_1 | [pid: 7|app: -1|req: -1/2] 10.243.191.131 () {44 vars in 683 bytes} [Tue Oct 5 00:06:39 2021] GET /favicon.ico => generated 21 bytes in 0 msecs (HTTP/1.1 500) 2 headers in 83 bytes (0 switches on core 0)

As described in 2. Connect to SSO Identity Provider (IDP)
I put ia.jwt and provider_info.json files to ./data/oidc/ but I don’t have and don’t know „client_secrets.json“.
It automatically creates an empty client_secrets.json after calling docker-compose up -d --build.

I think I may create the wrong ia.jwt file?
ia.jwt file

Best Regards,
Ya-Fan

Dear Ya-fan,

admittedly we don’t have an official contact to request these tokens. At the moment this would be either directly from me or via our contact form Plattform MaterialDigital (network and security architecture). Ontodocker should create the client_secrets.json from the IAT and the provider_info.json If a previous unseccsful attempt created an empty client_secrets.json, you should remove this.

Best

Joerg

Dear Jörg,

Thank you for your help. I’ve successfully installed Ontodocker along with Keycloak.
Since I was getting the error messages below, I decided to create the client_secrets.json file myself.

ontodocker_1  | Traceback (most recent call last):
ontodocker_1  |   File "./run.py", line 3, in <module>
ontodocker_1  |     ontoapp = create_app()
ontodocker_1  |   File "./app/__init__.py", line 45, in create_app
ontodocker_1  |     oidc.init_app(app)
ontodocker_1  |   File "/usr/local/lib/python3.7/dist-packages/flask_oidc/__init__.py", line 132, in init_app
ontodocker_1  |     secrets = self.load_secrets(app)
ontodocker_1  |   File "/usr/local/lib/python3.7/dist-packages/flask_oidc/__init__.py", line 196, in load_secrets
ontodocker_1  |     return _json_loads(open(content, 'r').read())
ontodocker_1  |   File "/usr/local/lib/python3.7/dist-packages/flask_oidc/__init__.py", line 51, in _json_loads
ontodocker_1  |     return json.loads(content)
ontodocker_1  |   File "/usr/lib/python3.7/json/__init__.py", line 348, in loads
ontodocker_1  |     return _default_decoder.decode(s)
ontodocker_1  |   File "/usr/lib/python3.7/json/decoder.py", line 337, in decode
ontodocker_1  |     obj, end = self.raw_decode(s, idx=_w(s, 0).end())
ontodocker_1  |   File "/usr/lib/python3.7/json/decoder.py", line 355, in raw_decode
ontodocker_1  |     raise JSONDecodeError("Expecting value", s, err.value) from None
ontodocker_1  | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
ontodocker_1  | unable to load app 0 (mountpoint='/') (callable not found or import error)
ontodocker_1  | *** no app loaded. going in full dynamic mode ***

I create the client_secrets.json like below:

{
    "web": {
        "issuer": "https://{KEYCLOAK_URL}/auth/realms/{REALM_NAME}",
        "auth_uri": "https://{KEYCLOAK_URL}/auth/realms/{REALM_NAME}/protocol/openid-connect/auth",
        "client_id": "{CLIENT_ID}",
        "client_secret": "{CLIENT_SECRET}",
        "token_uri": "https://{KEYCLOAK_URL}/auth/realms/{REALM_NAME}/protocol/openid-connect/token",
        "token_introspection_uri": "https://{KEYCLOAK_URL}/auth/realms/{REALM_NAME}/protocol/openid-connect/token/introspect"
    }
}

Then, it’s OK.

Best Regards,

Ya-Fan

1 Like

Thank you for sharing your solution. If you encounter further problems please continue posting your issues here. It might be helpful for other users.