DataSHIELD installation on Ubuntu - login into Opal fails

Support Beginner Support
1

For the installation I followed the instructions in the documentation:

https://opaldoc.obiba.org/en/latest/admin/installation.html

The software requirements are met:

openjdk version “1.8.0_312”

mysql Ver 15.1 Distrib 10.3.32-MariaDB

MongoDB shell version v3.6.8

R version 3.6.3

As the VM is a Ubuntu-20.04 distribution I simply installed the debian packages for agate, mica2, rock and opal:

apt-get install opal

apt-get install opal-python-client

apt-get install rock

apt-get install mica2

apt-get install mica-python-client

In R I installed the opalr package:

install.packages("opalr")

The configuration file I left unchanged, because this VM is currently only accessible internally via IP, not a public domain:

https://opaldoc.obiba.org/en/latest/admin/configuration.html

All services, agate, mica2, rock, mariadb, mongod and opal are running; at least according to their statis with:

systemctl status

From within R I can connect to the Rock-server and execute the example on Using R — Opal documentation, successfully.

There is one major problem: I can’t connect to OPAL via the web interface, though. The login page, http://192.168.100.94:8080/ui/index.html, always denies the authentication.

/var/log/opal/rest.log showed only “unknown” for user. Otherwise there is little insight from the logs:

{“@timestamp”:“2022-01-11T19:47:09.791+01:00”,“@version”:“1”,“message”:“/auth/sessions”,“logger_name”:“org.obiba.opal.web.security.AuditInterceptor”,“thread_name”:“qtp1522875398-31”,“level”:“WARN”,“level_value”:30000,“method”:“POST”,“ip”:“10.0.8.4”,“username”:“Unknown”,“status”:“403”}

I thought the default credentials are “administrator/password”. I tried to update the password with

dkpg-reconfigure opal

But any password entered there didn’t change the SHA256 entry in /etc/opal/shiro.ini. I then used

shiro-hasher -p password

to create the hash for “password” which pasted into /etc/opal/shiro.ini:

administrator=$shiro1$SHA-256$500000$TIBDu25yk3IYyUKgRJf2hQ==$Sd/ChI2QCqf4diT5iSVTbSZTt2V6XbNy6/xEiv8NC2k=,admin

I then restarted agate, mica2 and opal. As far as I understand authentication is down by agate, and shiro.ini is part of its configuration.

I still only got. when trying to log in. Capital “A” or not, i.e. “a” (“administrator”) didn’t make any difference either:

[image]

As far as I know, the data, and also user data is held in the mongodb database, right? /var/log/mongodb/mongodb.log didn’t show any connection attempts, though.

I connected on to the mongo shell on the console:

mongo --port 27017

There are no users or roles in the “agate” database defined:

use agate switched to db agate db.getRoles()

During what part of the installation are the user credentials in agate created? Doing dpkg-reconfigure agate allows to set a password for “administrator”. But there are still now users or roles in the agate database:

use agate; switched to db agate db.getUsers()

db.getRoles()

And login in the web interface also still fails.

2

Hi,

Do you have CSRF errors reported in the application log files (opal.log, mica.log, agate.log)?

By the way, your R version is really old: 3.6, then almost 2 years old, this can be a problem for some R packages. We also recommend also running Rock from a docker image (we provide some and you can build your own), it is much easier to maintain in a production environment.

Regards
Yannick

3

I found information about creating a CSRF exception in the browser for a site; I configured that in the client’s Chrome browser.

Also, I called the server’s login page with the internal host name, “opal.intern:8080”, instead of the 192.168.100.94, corresponding to the exception I defined in the Chrome.

This made no difference, though. The CSRF error still occurs in the log:

==> /var/log/opal/opal.log <== 2022-01-21 18:56:48,053 [qtp1522875398-34] WARN org.obiba.opal.web.security.CSRFInterceptor - CSRF detection: Host=opal.intern:8080, Referer=http://opal.intern:8080/ui/index.html

/var/log/opal/rest.log

{"@timestamp":"2022-01-21T18:56:48.053+01:00","@version":"1","message":"/auth/sessions","logger_name":"org.obiba.opal.web.security.AuditInterceptor","thread_name":"qtp1522875398-34","level":"WARN","level_value":30000,"method":"POST","ip":"10.0.8.4","username":"Unknown","status":"403"}
4

I see these CSFR errors in /var/log/opal/opal.log:

2022-01-21 18:40:53,418 [Timer-2] INFO  org.obiba.opal.r.service.RServerManagerService - R server 'rserver' added to cluster: default
2022-01-21 18:41:03,277 [qtp1280512370-27] WARN  org.obiba.opal.web.security.CSRFInterceptor - CSRF detection: Host=192.168.100.94:8080, Referer=http://192.168.100.94:8080/ui/index.html
2022-01-21 18:41:03,292 [qtp1280512370-27] WARN  org.obiba.opal.web.security.CSRFInterceptor - CSRF detection: Host=192.168.100.94:8080, Referer=http://192.168.100.94:8080/ui/index.html
2022-01-21 18:41:08,843 [qtp1280512370-118] WARN  org.obiba.opal.web.security.CSRFInterceptor - CSRF detection: Host=192.168.100.94:8080, Referer=http://192.168.100.94:8080/ui/index.html

The minimum requirements on Installation — Opal documentation require R version 3.6.3. Especially R is known to have packages that run only on certain versions of R (often minor updates break compatibility). Certain packages for the major 3.6 version have never been ported to the 4.x version.

In bioinformatics I had many bad experiences with that, we have to provide several R versions on the cluster.

If there are packages that require version 4.x, then I think it should be listed as the minimum required version in the requirements of the installation.

I can of course install another version of R, but I have to know which is the correct one.

The real problem is that the installation of the “opal”-Debian packages seem not to create a sufficient configuration for a complete environment, if there is no “administrator” user created in the MongoDB.

Opal version 4.3 was released, I upgraded to that by reinstalling the packages: apt-get update apt-get reinstall agate mica2 opal opal-python-client

That seems to have installed agate 2.53, mica2 4.6.5 and opal version 4.3.1. I then restarted the services:

systemctl restart agate; systemctl restart mica2; systemctl restart rock; systemctl restart opal

But the authentication problem persists, and also the CSRF errors. How can I fix those?

5

Hi,

CSRF protection was added recently and is a bit strict for now (and for sure would need some configuration for a more relaxed usage). You must access Opal through https (default port is 8443 for opal) when not using host with name localhost.

Regarding R, yes R 3.x is minimum but R 4.x is recommended. The limitation that I mentioned with R 3.x is not because of Opal, it is because of the R analysis that you might execute in the R server: it is up to you to anticipate your R needs. To help, we do provide ready-to-use R server images based on the latest R version (and related packages).

Regards
Yannick

6

Hi Yannick,

thank your for your quick response - after approval of my forum post. (In today’s video drop-in session I was told that this is a minor annoyance - it doesn’t slow me down, the demonstration project is not that urgent.)

Using the https-URL and port on the internal test server does allow me to log in. opal.intern:8443

And there is no more error message in the logs either.

I see about the R version. We are part of the “COMMITMENT”-project in Germany. We want to provide CAD and genomics data.

I don’t know, yet, what analyses and R scripts scientist want to execute on the data. I am working on the technical side providing the infrastructure.

On Ubuntu-20.04 I could update R to version 4.1 through the package manager. I have some familiarity with Docker, but I am struggling with persistency of data in Docker environments. So, I don’t feel like using it in a production environment.

Also, the OPAL-provided dockerfile seemed not to be up to date. There were problems connecting to the rock-server. Thus, we were advised by Hank Cao (Han.Cao@zi-mannheim.de) to use an installation on a classical VM.

Thank you for your help. I will report the progress to the scientists who will start using the installation with a sample project and importing data.

The next step will be to use a public domain and deploy it on our web server. As far as I understood, I have to make the appropriate adjustments in /etc/opal/opal-config.properties for that.

I hope that want be a problem, but as with all IT endeavours, I have feeling that I will post the odd question on this forum.

Thanks for providing that great support!

Best wishes,

Sven

7

Hi,

Yes, using https is mandatory when not accessing localhost, otherwise your username/password will travel in clear on the network.

You do not need to have opal in docker, just having the R server in docker is (much) more convenient (and you can have several different R servers on the same host, used by the same opal server). There is no data to be persisted in the R server, it is just a pure computation service, so it is perfectly fine to be flexible with these: opal is able to discover the R services at runtime.

As you will work on genetics, with ML apparently, you should consider having the R server(s) running on a separate host, for scalability. Also depending on the nature of the data, you will probably need to design appropriate resources.

The opal docker image is always up-to-date. Make sure your docker engine grabs the latest image for the tag you have chosen.

There is not much to be done in the opal-config.properties file to go live. Your efforts should concentrate on setting up a reverse proxy instead, with the appropriate security settings (and this is not opal specific).

For more advanced support (infrastructure design, data integration), please contact me for a commercial offer.

Regards
Yannick

1 Like