Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Remove file dependency #1754

Merged
merged 25 commits into from
Jan 15, 2020
Merged

Remove file dependency #1754

merged 25 commits into from
Jan 15, 2020

Conversation

larohra
Copy link
Contributor

@larohra larohra commented Jan 11, 2020

Description

Currently the Guest Agent relies on the filesystem to fetch cached data for the GoalState if the object is None. This behavior can lead to inconsistencies in the wire client objects as some thread might update the filesystems midway of execution of goal-state.

This PR removes the dependencies of the WireClient from the file system and keeps everything in memory. We still write to files because they help in debugging.

Issue #


PR information

  • The title of the PR is clear and informative.
  • There are a small number of commits, each of which has an informative message. This means that previously merged commits do not appear in the history of the PR. For information on cleaning up the commits in your pull request, see this page.
  • Except for special cases involving multiple contributors, the PR is started from a fork of the main repository, not a branch.
  • If applicable, the PR references the bug/issue that it fixes in the description.
  • New Unit tests were added for the changes made and Travis.CI is passing.

Quality of Code and Contribution Guidelines


This change is Reviewable

@codecov
Copy link

codecov bot commented Jan 11, 2020

Codecov Report

❗ No coverage uploaded for pull request base (develop@43ab21b). Click here to learn what that means.
The diff coverage is 78.57%.

Impacted file tree graph

@@            Coverage Diff             @@
##             develop    #1754   +/-   ##
==========================================
  Coverage           ?   68.66%           
==========================================
  Files              ?       82           
  Lines              ?    11708           
  Branches           ?     1641           
==========================================
  Hits               ?     8039           
  Misses             ?     3320           
  Partials           ?      349
Impacted Files Coverage Δ
azurelinuxagent/common/rdma.py 16.84% <0%> (ø)
azurelinuxagent/ga/monitor.py 89.27% <100%> (ø)
azurelinuxagent/common/singletonperthread.py 100% <100%> (ø)
azurelinuxagent/ga/update.py 88.46% <100%> (ø)
azurelinuxagent/common/protocol/wire.py 77.83% <66.66%> (ø)
azurelinuxagent/ga/remoteaccess.py 88.88% <80%> (ø)
azurelinuxagent/common/protocol/util.py 83.41% <85.1%> (ø)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 43ab21b...68cf77b. Read the comment docs.

azurelinuxagent/common/protocol/util.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Outdated Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
tests/protocol/test_wire.py Outdated Show resolved Hide resolved
tests/protocol/test_wire.py Show resolved Hide resolved
Copy link
Member

@narrieta narrieta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a few more comments... still reviewing

azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Outdated Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
Copy link
Member

@narrieta narrieta left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

a couple of comments... reviewing tests now

azurelinuxagent/common/rdma.py Show resolved Hide resolved
azurelinuxagent/ga/monitor.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
azurelinuxagent/common/protocol/wire.py Outdated Show resolved Hide resolved
tests/ga/test_update.py Show resolved Hide resolved
@larohra larohra requested a review from narrieta January 13, 2020 21:24
narrieta
narrieta previously approved these changes Jan 13, 2020
Copy link
Contributor

@pgombar pgombar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One file dependency should be removed; other comments are nits.

self.ext_conf = ExtensionsConfig(xml_text)
return self.ext_conf
raise ProtocolError("Trying to fetch Extension Conf before initialization!")
return self._ext_conf

def get_ext_manifest(self, ext_handler, goal_state):
local_file = MANIFEST_FILE_NAME.format(ext_handler.name, goal_state.incarnation)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update to remove file dependency?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I didnt get what you meant. If you meant MANIFEST_FILE_NAME this file, then this is only being used to save the local cache, its never read from.

    def get_ext_manifest(self, ext_handler, goal_state):
        local_file = MANIFEST_FILE_NAME.format(ext_handler.name, goal_state.incarnation)
        local_file = os.path.join(conf.get_lib_dir(), local_file)

        try:
            xml_text = self.fetch_manifest(ext_handler.versionUris)
            self.save_cache(local_file, xml_text)
            return ExtensionManifest(xml_text)
        except Exception as e:
            raise ExtensionDownloadError("Failed to retrieve extension manifest. Error: {0}".format(ustr(e)))

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this could apply for the get_gafamily_manifest case as well.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah sorry, the name of the method (implying a getter) and the fact that we're reading something from a file made me think this was retrieving state from a file.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this could apply for the get_gafamily_manifest case as well.

Yes it would, both cases its the same, its just being used to save cache only

    def get_gafamily_manifest(self, vmagent_manifest, goal_state):
        local_file = MANIFEST_FILE_NAME.format(vmagent_manifest.family, goal_state.incarnation)
        local_file = os.path.join(conf.get_lib_dir(), local_file)

        try:
            xml_text = self.fetch_manifest(vmagent_manifest.versionsManifestUris)
            fileutil.write_file(local_file, xml_text)
            return ExtensionManifest(xml_text)
        except Exception as e:
            raise ProtocolError("Failed to retrieve GAFamily manifest. Error: {0}".format(ustr(e)))

azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
tests/protocol/test_wire.py Outdated Show resolved Hide resolved
self.ext_conf = ExtensionsConfig(xml_text)
return self.ext_conf
raise ProtocolError("Trying to fetch Extension Conf before initialization!")
return self._ext_conf

def get_ext_manifest(self, ext_handler, goal_state):
local_file = MANIFEST_FILE_NAME.format(ext_handler.name, goal_state.incarnation)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And this could apply for the get_gafamily_manifest case as well.

@@ -62,7 +62,12 @@ def run(self):
if self.incarnation != current_incarnation:
# something changed. Handle remote access if any.
self.incarnation = current_incarnation
self.remote_access = self.protocol.client.get_remote_access()
try:
self.remote_access = self.protocol.client.get_remote_access()
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is what we discussed in the meeting - that this code would be reverted. Correct?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope not this code, that was the RDMA code. In this I added a log.warn incase there was a ProtocolError but reverted that change because this file in majorly empty

pgombar
pgombar previously approved these changes Jan 15, 2020
azurelinuxagent/common/protocol/wire.py Show resolved Hide resolved
@larohra
Copy link
Contributor Author

larohra commented Jan 15, 2020

remote_access = RemoteAccess(xml_text)
return remote_access
if self._remote_access is None:
raise ProtocolError("Trying to fetch Remote Access before initialization!")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems like the previous code was returning None rather than raising

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I was initially just setting it to None too but changed to raise a ProtocolError to keep it at par with the rest of the code. I am handling this now in the remoteaccess.py file

                    try:
                        self.remote_access = self.protocol.client.get_remote_access()
                    except ProtocolError:
                        self.remote_access = None

@larohra larohra merged commit f38cede into Azure:develop Jan 15, 2020
@larohra larohra deleted the RemoveFileDependency branch January 15, 2020 21:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants