Datashield R API

Support Developer support
21

I cannot reproduce, maybe restart the R session and start the script with:

library(DSLite)

There are also setup shortcut functions, for instance the one of DASIM:


library(DSLite)

DSLite::setupDASIMTest()

ls()

conns <- datashield.login(logindata.dslite.dasim, assign=T)
22

:frowning: I have restarted, cleared all variables and tried the shortcut functions and still get the same error. Any other ideas?

23

  • Can you make sure that the opal package is unloaded ? because datashield.login function should be the one of DSI package.

  • What do you have in the logindata.dslite.dasim?

  • Do you have R6 package installed (required by DSLite)?

24

Opal package is uninstalled (though previously it was installed)

R6 is loaded

logingdata.dslite.dasim

is

server url user password table options driver
1 sim1 dslite.server DASIM1 DSLiteDriver
2 sim2 dslite.server DASIM2 DSLiteDriver
3 sim3 dslite.server DASIM3 DSLiteDriver

I’m trying a fresh install too

25

I know what’s the issue, I can reproduce it now: you do not have any server-side datashield package installed locally. The command dslite.server$config() will probably report empty DS settings.

You can install at least the dsBase package (DSI branch) using:

devtools::install_github("datashield/dsBase", ref="DSI")
26

Yes… that has done it. I’m still trying to work out in my head how this new set up works, hence I hadn’t realised the need for the server side packages. Thanks for your patience.

27

Actually having server side packages installed is not an absolute requirement. I have even designed the DSLiteServer in a way that the datashield configuration could be setup manually (there are functions for defining agggregate and assign methods). It’s just that I did not test the case of login with an empty DS configuration (not a DSLite specific issue by the way)… Should not be a problem to fix.

The README can help with understanding the DSLite architecture.

28

Got mine working too. Thanks!

29

What do you think the migration path to DSI would be? Given you’ve ported many (all?) of the functions to it anyway (e.g. https://github.com/datashield/dsBaseClient/tree/DSI) do you think it is just a case of merging with the master branch, or is it going to be a breaking change for some people?

I was planning on waiting for the testing to be done first so we can convince ourselves DSI is working, but you’ve done a lot much faster than I thought it would happen! Maybe we need to reconsider this and move to it much sooner.

30

Hi Olly,

I have created a DSI branch for all of these DataSHIELD packages:

  • dsBetaTestClient
  • dsBetaTest
  • dsBaseClient
  • dsBase
  • dsGraphicsClient
  • dsModellingClient
  • dsModelling
  • dsMxClient
  • dsStatsClient

There are changes in server side packages as well, which I did not expect, but these are only required for DSLite because when using environment-dependent functions like eval, get, assign and as.formula, one must specify in which environment the symbols are to be looked for or assigned to. By default it ends in the Global Env but with DSLite, there is no Global Env for the server side execution environment (because the Global Env is the client side and we do not want them to overlap). For instance dsBetaTest is using a lot of eval() calls. The modification is easy though (adding parameter envir = parent.frame()).

Then if you want to give it a try, install both client and server sides packages from the DSI branch (there is not necessarily one for the server side). All the testthat tests are passing successfully.

In other words, the DSI migration is done at 90%, what remains to be done is testing with your pipeline.

For the end users, the only changes I expect are:

  • installing DSOpal (with dependencies DSI and opalr) on client side
  • adding library(DSOpal) at the beginning (and removing library(opal))
  • for the ones using the opal-specific functions that have been added in dsBetaTestClient, replace by the corresponding DSI functions (the ones starting with DSI::datashield.connections* for instance).

Once I have your feedback and OK, I can start the submission of the DSI, DSOpal and DSLite packages in the official CRAN. This can be done before the datashield analysis packages are ready. I have already gone through the CRAN submission procedure for opalr, they are quite picky regarding the quality of the packaging, approval can take several weeks, but it’s worth the effort in terms of visibility and credibility.

Cheers
Yannick

31

Thanks Yannick.

We are going to have a local meeting about this this week to see how we go forward. I’ll feed back to you about that.

For the official CRAN - I have only looked briefly at the idea of submitting DS to it, but I came to the conclusion it would be hard as the client-server pairing would make the mandatory testing difficult to demonstrate. Am I incorrect in thinking that? In which case what would we gain from putting DSI, DSOpal and DSLite on the official CRAN? (I am still a way off understanding all three of those so I may be missing something obvious!).

Cheers, Olly

32

DSI (=the API), DSOpal and DSLite (=some implementations of the API) are the foundations of the DataSHIELD infrastructure on the R client side. As such these are expected to be very stable, and will be used directly by mainly the data repository integrators (Obiba and Molgenis teams) and also by the analysis package developers (Datashield team). Having them in the official R ecosystem will bring some visibility to DataSHIELD, will enforce good packaging practices (not so obvious!) and will simplify the setup of the dev environment in Rstudio.

For the DS analysis packages that you are developing, DSLite should make things easier for including package tests and documentation examples (but still, both client and server packages are to be installed locally). I am not sure it’s worth having them in the official CRAN, as updates are expected to not happen too frequently (1-2 months…) because the CRAN is managed by volunteers. If there is a security fix to be deployed, it’s better if you control your own CRAN.

Yannick

33

What about DataShield Gateway? So it can become DS.Gateway?

P.

34

Hi,

I am pleased to announce that the first versions of the DataSHIELD R API packages are available in the official CRAN repository:

Thanks to the CRAN maintainers who reviewed the code and the documentation, the quality of these packages is expected to be good.

These releases should be used as the reference for the next dsBase and dsBaseClient version.

Cheers
Yannick