Bug - Exported event schema validation fails

[imranmmd17]

PyMISP exported event fails schema validation.

event.to_json() returned event is when passed through the standard MISP schema (https://github.com/MISP/MISP/blob/2.4/format/2.4/schema.json), the schema validation fails for:

1. publish_timestamp - is expected to be str
2. tags - Event and Attribute tag's local attribute type is expected to be int

[imranmmd17]

@Rafiot - Can you please look into it. Thanks.

[Rafiot]

The events generated by PyMISP work just fine on MISP, so it is probably more an issue with the schema itself.

[Rafiot]

ok, update on that: PyMISP to_json returns a blob that is compatible with what MISP expects (float or int) when the schema says string.

We could consider to add a method that casts the int/float to str, but the default will stay the same for backward compatibility reasons. What is your use-case?

[imranmmd17]

Thanks @Rafiot. Our requirement is to export the MISP event and ensure it passes the MISP standard schema before passing it on to the data pipeline for subsequent processing. At every stage of the pipeline, the very thing they do is schema validation again.

[Rafiot]

Alright, I started to look at that:

1. will require more detailed investigation as it is converted to an integer at a place that makes it somewhat hard to change. nevermind, fixed: 1d24c1b
2. the local key in tag isn't in the schema at all, can you give more details on how you got the warning, please?

[imranmmd17]

1. Thanks @Rafiot for fixing the timestamp. LGTM.
2. When an event is downloaded using MISP WebUI, the tag's local attribute is number (i.e. int). Whilst the API event.to_json() returns it as str. Yes the local key isn't in the schema, however we've added it for our use case to match with the WebUI downloaded event. We expect the outcome to be the same whether an event is d/l using WebUI or using API. It's not a show stopper, but good to have consistency.

[dragsu]

@Rafiot I want to extend this. What is the reason for the blanket _int_to_str(to_return) call inside to_dict method in AbstractMISP.py? As a result, Tag's local values (0 and 1) get changed into string values ("0" and "1") , which is unexpected.

P.S
I have added a fix for this #1171

[Rafiot]

It is a bug on MISP side, that should be handled as a boolean, and not 0/1. It will be fixed in MISP asap.

[iglocska]

FYI, the fix is now on develop and will go into the release later today.

[dragsu]

MISP v2.4.187 got an upstream fix for this. Now local values are represented as true and false.