ERROR: Could not upload pipelet: b'{"error": "{\\"error\\": \\"Unknown pipelet loading error\\"}"}'

Training with Squirro
1

Hi,

When I try to upload the pipelet from the ‘pipelet’ lession, I get the following error:
Uploading 0 data files
ERROR: Could not upload pipelet: b’{“error”: “{\“error\”: \“Unknown pipelet loading error\”}”}’

How can I solve this? I am using Squirro SDK version 3.8

2

Hi TalitaAnthonio,
Usually this occurs when there is some sort of validation error or misconfiguration in your pipelet.
Could you share the output of: pipelet -vvvv validate <your pipelet name>.py as well as the output of the upload command with the -vvvv flag?

It would look something like:

pipelet -vvvv upload --token <your_token> --cluster <cluster> pipelet.py "Pipelet Name"

Thank you!
Aaron

1 Like
3

Hi Aaron,

For Pipelet 1: When I use validate, I get OK: Validated and loaded.
For Pipelet 2; this pipelet should read a pdf (like in the tutorial). If I do validate, I get the error No module named ‘squirro.lib.storage’.

For both pipelets I get the following log when I try to upload (where CLUSTER = my cluster)

2023-09-07 14:47:38,518 setting up squirro.sdk
2023-09-07 14:47:38,518 registering JSON module 'json'
2023-09-07 14:47:38,519 skipping database endpoint discovery for service sdk
2023-09-07 14:47:38,519 skipping redis endpoint discovery for service sdk
2023-09-07 14:47:38,519 skipping redis endpoint discovery for service sdk
2023-09-07 14:47:38,522 Starting new HTTP connection (1): CLUSTER:80
2023-09-07 14:47:38,579 CLUSTER:80 "POST /api/user/oauth2/token HTTP/1.1" 200 None
2023-09-07 14:47:38,582 Uploading 0 data files
2023-09-07 14:47:38,616 Starting new HTTP connection (1): CLUSTER:80
2023-09-07 14:47:38,690 CLUSTER:80 "PUT /api/topic/v0/squirro/pipelets/Readability H
TTP/1.1" 400 59
2023-09-07 14:47:38,691 ERROR: Could not upload pipelet: b'{"error": "{\\"error\\": \\"Unknown pipelet loading err
or\\"}"}'

Thank you so much!

1 Like
4

Okay, so its most likely to do with the instance failing to import a custom library which is not present on your cloud instance.

A good place to check this is on the server logs located at /var/log/squirro/plumber/plumber.log
You should see an error thrown there.

You can check this by SSHing into your instance:

$ ssh username@your_host.squirro.cloud

then to check the log files:

[user@host] $ less /var/log/squirro/plumber/plumber.log

or you may be able to catch it as it happens by executing squirro_tail_errors and attempting to upload again.

[user@host] $ squirro_tail_errors

Another option is to check which modules are already installed on the server python environment by logging onto the remote server as shown above, then running:

[user@host] $ squirro_activate

and then listing the packages with

[(virtualenv38) user@host] $ pip list | column

This should give you a list of all the packaged you have on available. You can verify if you are importing any packaged not listed there.

If this is the case, you can install the packages in the squirro environment by sqitiching your user to root and activating squirro again, then:

 [(virtualenv38) root@host ]# pip install <package-name>

Otherwise if you could paste the errors you see related to the pipelet here or DM me them, I can look into it further.

Hope this helps,
Aaron

1 Like