ErrorAccessDenied thrown if folder contains private items

[PyTomE]

Hi,
maybe I tried it the wrong way around, maybe I just stuck somewhere. However, I try to connect to an exchange server in order to extract several calendars.
In general this works, thanks to your great tool, but now I try the following:
I have a generic user, say "generic" which can access others calendars whenever they grant this user access within Outlook.
This works until a user sets an appointment to "private", which shouldn't be a huge problem but...
...

account = Account(primary_smtp_address=user_account, config=config,
                      autodiscover=False, access_type=DELEGATE)
items = account.calendar.view(start=startdate, end=enddate) # Filter by a date range
for item in items.all().order_by('start'): # no matter if order_by is used or not
    ... do something...

Traceback (most recent call last):
  File "getCalendar.py", line 109, in <module>
    for item in items.all().order_by('start'):
  File "e:\progs\Continuum\Anaconda2\lib\site-packages\exchangelib\queryset.py", line 197, in __iter__
    for val in result_formatter(self._query()):
  File "e:\progs\Continuum\Anaconda2\lib\site-packages\exchangelib\queryset.py", line 163, in _query
    items = sorted(items, key=f.field_path.get_value, reverse=f.reverse)
  File "e:\progs\Continuum\Anaconda2\lib\site-packages\exchangelib\account.py", line 534, in fetch
    for i in GetItem(account=self).call(items=ids, additional_fields=additional_fields, shape=IdOnly):
  File "e:\progs\Continuum\Anaconda2\lib\site-packages\exchangelib\services.py", line 456, in _pool_requests
    for elem in elems:
  File "e:\progs\Continuum\Anaconda2\lib\site-packages\exchangelib\services.py", line 283, in _get_elements_in_response
    container_or_exc = self._get_element_container(message=msg, name=self.element_container_name)
  File "e:\progs\Continuum\Anaconda2\lib\site-packages\exchangelib\services.py", line 256, in _get_element_container
    self._raise_errors(code=response_code, text=msg_text, msg_xml=msg_xml)
  File "e:\progs\Continuum\Anaconda2\lib\site-packages\exchangelib\services.py", line 273, in _raise_errors
    raise vars(errors)[code](text)
exchangelib.errors.ErrorAccessDenied: Access is denied. Check credentials and try again.

I tried to dive into the code but it looks as if exchangelib tries to inspect the elements in the queryset too much and fails while creating the iterator.
Thus I found no way to circumvent that problem but rewriting the services-module (didn't do that yet).
Is there anything I missed yet? Or a better way to handle this?

Thanks in advance
Thomas

[ecederstrand]

I agree this is not ideal. The assumption has been that ErrorAccessDenied is a general error that would happen for all items returned, but apparently that's not the case.

You could try to allow this error in the response instead of raising it. Try adding this at the top of your code:

from exchangelib.services import GetItem
from exchangelib.errors import ErrorAccessDenied
GetItem.ERRORS_TO_CATCH_IN_RESPONSE += (ErrorAccessDenied,)

This should result in the private items being returned as ErrorAccessDenied instances when you iterate over the results of the query, instead of ErrorAccessDenied being raised directly. You can then decide to raise the exception instance or ignore it, as appropriate for your use case.

If this works for you, I'll add a permanent fix in the code.

[PyTomE]

Yes, that solves my issue in case that I do the loop over
for item in items.all(): ...
in case of
for item in items.all().order_by('start'):...
it still returns an error that makes sense only in a limited way since start and end times of appointments are visible even if the appointment is private. Of course not for ErrorAccessDenied...

Traceback (most recent call last):
  File "getCalendar.py", line 112, in <module>
    for item in items.all().order_by('start')::
  File "e:\progs\Continuum\Anaconda2\lib\site-packages\exchangelib\queryset.py", line 197, in __iter__
    for val in result_formatter(self._query()):
  File "e:\progs\Continuum\Anaconda2\lib\site-packages\exchangelib\queryset.py", line 163, in _query
    items = sorted(items, key=f.field_path.get_value, reverse=f.reverse)
  File "e:\progs\Continuum\Anaconda2\lib\site-packages\exchangelib\fields.py", line 125, in get_value
    return getattr(item, self.field.name)
AttributeError: 'ErrorAccessDenied' object has no attribute 'start'

But this way I have at least a workaround and can go further. Thanks

[ecederstrand]

Ah, I think I know what's going on here.

We first call FindItem to list all items in a folder. This is fine since private items are allowed to be listed there, with the private fields properly anonymized.

Then we call GetItem to fetch all fields for the items returned by FindItem. This fails for private items because we, among other fields, request the body field, which is private.

You can test if this is indeed the case by changing your query to only fetch non-private fields:

for item in items.all().only('start', 'end', 'sensitivity').order_by('start'):
    print(item.start, item.end, item.sensitivity)

Finally, you're getting the AttributeError because we try to apply sorting even on items that are in fact exceptions. This needs to be fixed.

[PyTomE]

Oh yes, that did the final trick!
I even can populate the only-list by those attributes I really need and this makes it not only work, it makes it faster too. ...Of course, since I don't dig so deep into the appointments.
Now, I can proceed with my work. Thanks a lot again...

[ecederstrand]

Great, thanks for the clarification!

I'm not sure how to make a more general fix where some_folder.all() will just work when the folder contains private items. With EWS, we can only specify the set of returned fields on a per-request basis, but then we can't get all items in one go, because we want one set of fields for public items, and another for private items. Maybe you just have to do something like:

public_items = some_folder.exclude(sensitivity='Private')
private_items = some_folder.filter(sensitivity='Private').only(list_of_public_fields)

But I have no idea which fields on a private item will throw ErrorAccessDenied if you try to fetch them anyway. We'll need to find the correct values for list_of_public_fields, possibly by trial-and-error, before we can proceed with this issue.

[driesken]

I think I'm having the same issue when looping over public and private calendaritems when doing this:
for item in account.calendar.view(start=start_dt, end=end_dt, max_items=5):

2018-09-19 15:01:44,929 DEBUG GetItem._get_elements result 1 of 1 is ready
2018-09-19 15:01:44,929 INFO The specified object was not found in the store., Item not found.
2018-09-19 15:01:44,929 ERROR list index out of range

Is it possible to skip the private items or use something like a filter/exclude with account.calendar.view?

[ecederstrand]

My previous comment in this thread describes how to filter private and public items.

Unfortunately, you cannot combine view() with filter(), so you'll have to do this in two steps:

private_items = []
public_items = []
for m in a.calendar.view(start=start, end=end).only('sensitivity'):
    if m.sensitivity == 'Private':
        private_items.append(m)
    elif m.sensitivity == 'Normal':
        public_items.append(m)

full_public_items = list(a.fetch(public_items))
# May need to select public-only fields to avoid errors
full_private_items = list(a.fetch(private_items, only_fields=[...]))

[ecederstrand]

We really need a list of private fields before we can work on this issue. I cannot reproduce locally. If you stumble across this issue, try running this on a shared calendar that contains private items:

from exchangelib import Account
from exchangelib.errors import ErrorAccessDenied

a = Account(...)
for f in a.calendar.allowed_item_fields(version=a.version):
    try:
        a.calendar.filter(sensitivity='Private').only(f.name)[0]
    except ErrorAccessDenied:
        print(f.name, 'is a private field')