Hi, when I enter https://srv1.fedana.cloud in the browser, I get the error: “# 502 Bad Gateway — nginx/1.18.0”
The server worked fine a several weeks ago, docker processes look fine, nginx works fine, there is plenty of free memory. How do I fix this?
Kr, Wilmar
Hi,
Are you using opal or armadillo?
Is opal/armadillo application running?
Regards
Yannick
OPAL, the whole DataSHIELD stack is running in docker containers using the yml file from the OPAL documentation (sort of).
Then the next question is: what do you have in the log file? (opal.log)
Dear Yannick,
I have decided to cut the Gordian knot by reinstalling the server following the tutorial [1], which takes about 30min. Works again!
Thanks, Wilmar
@yannick Ok, it worked for while, but when I tried importing data, and I again got “502 Bad Gateway” error, when trying to login to OPAL on the website.
In additon, when trying to connect with opal.login() I get now the error: “Error: [Client error: (403) Forbidden]”
I guess, I have to go to the bottom of this. Where do I find opal.log when running datashield as a docker image?
Kr, Wilmar
@yannick I have found the opal.log, last lines (from FileUploadException):
ng,javax.ws.rs.core.UriInfo,javax.servlet.http.HttpServletRequest) throws java.io.IOException,org.apache.commons.fileupload.FileUploadException]
2023-06-01 10:54:00,165 [qtp769896530-42] INFO org.obiba.opal.web.FilesResource - The following file was uploaded to Opal file system : file:///home/administrator/R/4488375/file4adc73071152.rds
2023-06-01 10:54:00,170 [qtp769896530-42] INFO org.obiba.opal.core.service.security.realm.SpatialRealm - Clear cache perms for USER:administrator
2023-06-01 10:54:13,443 [qtp769896530-163] WARN org.obiba.opal.spi.r.datasource.magma.RVariableHelper - Not a valid ‘opal.value_type’ attribute value: character
2023-06-01 10:54:13,492 [qtp769896530-163] INFO org.obiba.magma.support.AbstractValueTable - Entity batch size for c19vxcfs.am: 100000
2023-06-01 10:54:13,655 [pool-15-thread-1] INFO org.obiba.opal.shell.service.impl.DefaultCommandJobService - Starting task 1
2023-06-01 10:54:13,661 [pool-15-thread-1] INFO org.obiba.opal.core.service.ProjectsState - Project c19vxcfs to state BUSY
2023-06-01 10:54:13,739 [pool-15-thread-1] INFO org.obiba.magma.support.DatasourceCopier - Copying ValueTable ‘c19vxcfs.am’ to ‘c19vxcfs.c19vxcfs.am’ (copyMetadata=true, copyValues=false).
2023-06-01 10:55:20,967 [Index Synchronization Consumer org.obiba.opal.search.IndexSynchronizationManager$SyncConsumer@1653011a] INFO org.obiba.es.opal.support.ESIndexManager - Updating index mapping [opal-variables] for c19vxcfs__c19vx
cfs-am
2023-06-01 10:55:21,116 [elasticsearch[Vague][clusterService#updateTask][T#1]] INFO org.elasticsearch.cluster.metadata - [Vague] [opal-variables] update_mapping [Variable]
2023-06-01 10:55:21,478 [elasticsearch[Vague][clusterService#updateTask][T#1]] INFO org.elasticsearch.cluster.metadata - [Vague] [opal-variables] update_mapping [Variable]
2023-06-01 10:57:18,303 [SessionValidationThread-1] INFO org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Validating all active sessions…
2023-06-01 10:57:18,304 [SessionValidationThread-1] INFO org.apache.shiro.session.mgt.AbstractValidatingSessionManager - Finished session validation. No sessions were stopped.
Ok, you need to troubleshoot a bit:
I do not understand if the file was uploaded? FYI I generated a dataset same count of rows and columns as yours and it is only 8MB in RDS format (data are numbers) and it works. What kind of data do you have?
The memory of the server does not tell how much is allocated to opal (default is 2G). You can check opal memory usage in Administration > Java Virtual Machine. You can allocate more memory to opal using the JAVA_OPTS argument
Have you checked the R server logs (that you can get from Administration > R)? it may be informative
@yannick Rescaled server from 4GB to 8GB RAM, Opal web works again, but still cannot import a file (1676488x2) with with object.size of 120 MB.
opal.table_save() got stuck at 83% or fails with “Request entity too large”, file was uploaded with only 15,000 rows. Data object size is 120MB in R memory. Data had mixed 1x character and 1x numeric data (1000 rows works, 10000 rows works, 100000 rows works, but 1 000 000 rows fails, “Request entity too large”).
- JAVA_OPTS=-Xms1G -Xmx8G -XX:+UseG1GC
Seems alright for a server with 8GB RAM?
Download: Nextcloud
Thanks for reporting these numbers; that will help.
If you allow opal to use all the memory of the server (8G) you’ll get into trouble when the R server (or any other process, like the database or even the system) needs some room!
The “Request entity too large” is probably coming from your nginx reverse proxy, see Default nginx client_max_body_size. Opal has a similar limit too, but it is 500M by default, which should be fine in your case.
This being said, given the size of the dataset and the context of usage being Datashield, you should use the “resource” approach instead: see How to NOT Import Data. You could, for instance, upload the prepared RDS (or TSV) file on the server at a location that is accessible by the R server, or you could upload it in a SQL database (mysql, postgresql). Then you declare in Opal how to connect to this data source, and Opal will just forward this information to R on datashield.assign.resource request; the R server will then take care of the connection. This would keep the opal process memory usage to a minimum and reduce the data transfers.
Regards
Yannick
Hi @yannick , I have exported the data as tibble frame to an RData file and moved it onto the server in the local docker filesystem (/home/dsuser/opal/fs/home/dsuser/data/*.rda), and added a resource manually via OPAL GUI (category: local, Type: R Data file - local, Path: as above, R object class: tbl_df).
On the client I can see the resource, but then I get errors:
datashield.assign.resource(conns=connections, symbol = “rsam”, resource=“c19vxcfs.c19vxcfsam” )
Assigned all resource (rsam ← …) [==================================================] 100% / 1s
datashield.assign.expr(conns=connections, symbol = “expam”, expr=quote(as.resource.data.frame(rsam, strict = TRUE)))
Assigned expr. (expam ← as.resource.data.frame(rsam, strict = TRUE)) [================] 100% / 0s Error: There are some DataSHIELD errors, list them with datashield.errors()
datashield.errors()
$server1 [1] “Command ‘as.resource.data.frame(rsam, strict = TRUE)’ failed on ‘server1’: Error while evaluating ‘is.null(base::assign(‘expam’, value={resourcer::as.resource.data.frame(rsam, strict = TRUE)}))’ → Error in readChar(con, 5L, useBytes = TRUE) : cannot open the connection\n”
What is the problem?
Kr, Wilmar
Hi,
You have chosen to create a local file resource. This type of resource requires that the file be accessible by the R server (not opal). Considering that your R server is running in a Docker container, by default the files from the host are not accessible: a volume must be declared in the docker compose file to gain this access.
For instance:
rock:
image: datashield/rock-base:latest
volumes:
- /some/host/path:/srv
...
In this example you will upload the file on the host at:
/some/host/path/data/myfile.Rdata
and you will declare the resource’s local path:
/srv/data/myfile.Rdata
Regards
Yannick
Dear @yannick, thanks so much, works now! I have added the description of installilng and using “resources” to my tutorial at Federated Analysis with R/DataSHIELD – Installation, Data Import, Data Analysis [Tutorial] – Wilmar Igl, PhD.
Kr, Wilmar